Continue-Files-Glob.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## I've been using OS to read every file in a folder\n",
    "\n",
    "`imoirt os`\n",
    "\n",
    "`folder = 'folder_name'`\n",
    "`for singel_file in os.listdirs(folder):`\n",
    "    `with open(os.path.join(folder,single_file)) as file:`      \n",
    "\n",
    "\n",
    "\n",
    "## lets use glob instead ;) \n",
    "**unix style pathname pattern expansion**<br>\n",
    "using glob patterns to specify set of file names! which uses **wildcard chars**<br>\n",
    "`glob.glob(pathname)` will return a **list** of path names that match pathname;)<br>\n",
    "lets see!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- will find all pathnames that end in .txt in my folder (this case all of them, 88 files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reviews/26-dunkirk-2017-film.txt\n",
      "Reviews/57-army-of-shadows.txt\n",
      "Reviews/48-alien-film.txt\n",
      "Reviews/43-bride-of-frankenstein.txt\n",
      "Reviews/54-the-400-blows.txt\n",
      "Reviews/63-manchester-by-the-sea-film.txt\n",
      "Reviews/5-mad-max-fury-road.txt\n",
      "Reviews/46-wonder-woman-2017-film.txt\n",
      "Reviews/49-bicycle-thieves.txt\n",
      "Reviews/19-laura-1944-film.txt\n",
      "Reviews/98-toy-story.txt\n",
      "Reviews/90-on-the-waterfront.txt\n",
      "Reviews/39-toy-story-3.txt\n",
      "Reviews/82-tokyo-story.txt\n",
      "Reviews/23-la-grande-illusion.txt\n",
      "Reviews/92-the-last-picture-show.txt\n",
      "Reviews/36-rear-window.txt\n",
      "Reviews/78-gone-with-the-wind-film.txt\n",
      "Reviews/60-a-streetcar-named-desire-1951-film.txt\n",
      "Reviews/84-pinocchio-1940-film.txt\n",
      "Reviews/32-king-kong-1933-film.txt\n",
      "Reviews/58-arrival-film.txt\n",
      "Reviews/64-dr.-strangelove.txt\n",
      "Reviews/40-argo-2012-film.txt\n",
      "Reviews/22-a-hard-day27s-night-film.txt\n",
      "Reviews/37-selma-film.txt\n",
      "Reviews/83-hell-or-high-water-film.txt\n",
      "Reviews/30-gravity-2013-film.txt\n",
      "Reviews/55-logan-film.txt\n",
      "Reviews/10-metropolis-1927-film.txt\n",
      "Reviews/42-the-big-sick.txt\n",
      "Reviews/72-rosemary27s-baby-film.txt\n",
      "Reviews/53-12-angry-men-1957-film.txt\n",
      "Reviews/4-get-out-film.txt\n",
      "Reviews/11-e.t.-the-extra-terrestrial.txt\n",
      "Reviews/44-zootopia.txt\n",
      "Reviews/27-the-maltese-falcon-1941-film.txt\n",
      "Reviews/52-up-2009-film.txt\n",
      "Reviews/1-the-wizard-of-oz-1939-film.txt\n",
      "Reviews/89-apocalypse-now.txt\n",
      "Reviews/9-the-godfather.txt\n",
      "Reviews/75-the-wrestler-2008-film.txt\n",
      "Reviews/100-battleship-potemkin.txt\n",
      "Reviews/94-the-grapes-of-wrath-film.txt\n",
      "Reviews/12-modern-times-film.txt\n",
      "Reviews/77-l.a.-confidential-film.txt\n",
      "Reviews/17-moonlight-2016-film.txt\n",
      "Reviews/35-rashomon.txt\n",
      "Reviews/80-skyfall.txt\n",
      "Reviews/16-casablanca-film.txt\n",
      "Reviews/3-the-third-man.txt\n",
      "Reviews/34-the-adventures-of-robin-hood.txt\n",
      "Reviews/45-m-1931-film.txt\n",
      "Reviews/69-the-babadook.txt\n",
      "Reviews/21-snow-white-and-the-seven-dwarfs-1937-film.txt\n",
      "Reviews/62-star-wars-the-force-awakens.txt\n",
      "Reviews/87-star-trek-film.txt\n",
      "Reviews/97-jaws-film.txt\n",
      "Reviews/29-12-years-a-slave-film.txt\n",
      "Reviews/85-the-jungle-book-2016-film.txt\n",
      "Reviews/73-finding-nemo.txt\n",
      "Reviews/8-inside-out-2015-film.txt\n",
      "Reviews/93-harry-potter-and-the-deathly-hallows-part-2.txt\n",
      "Reviews/51-the-treasure-of-the-sierra-madre-film.txt\n",
      "Reviews/31-sunset-boulevard-film.txt\n",
      "Reviews/99-the-godfather-part-ii.txt\n",
      "Reviews/7-all-about-eve.txt\n",
      "Reviews/61-the-night-of-the-hunter-film.txt\n",
      "Reviews/66-vertigo-film.txt\n",
      "Reviews/38-taxi-driver.txt\n",
      "Reviews/91-the-wages-of-fear.txt\n",
      "Reviews/33-spotlight-film.txt\n",
      "Reviews/25-the-battle-of-algiers.txt\n",
      "Reviews/86-la-la-land-film.txt\n",
      "Reviews/2-citizen-kane.txt\n",
      "Reviews/50-seven-samurai.txt\n",
      "Reviews/74-brooklyn-film.txt\n",
      "Reviews/18-psycho-1960-film.txt\n",
      "Reviews/15-boyhood-film.txt\n",
      "Reviews/79-the-good-the-bad-and-the-ugly.txt\n",
      "Reviews/67-the-dark-knight-film.txt\n",
      "Reviews/68-touch-of-evil.txt\n",
      "Reviews/59-baby-driver.txt\n",
      "Reviews/96-man-on-wire.txt\n",
      "Reviews/41-toy-story-2.txt\n",
      "Reviews/20-nosferatu.txt\n",
      "Reviews/14-singin-in-the-rain.txt\n",
      "Reviews/6-the-cabinet-of-dr.-caligari.txt\n"
     ]
    }
   ],
   "source": [
    "for single_file in glob.glob('Reviews/*.txt'): #this is the wildcard 'Reviews/*.txt'\n",
    "    print(single_file)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What if I want to open them? \n",
    "#### it is so important to use encode parameter whenever reading a file.. <br> to correctly decode the unicode and preventing errors \n",
    "The actual encoding depends on the source of the text, typically in web its utf8;<br>\n",
    "Just inspect one of the web files and check: charset = 'utf-8'? :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dunkirk (2017)\n",
      "http://www.rogerebert.com/reviews/dunkirk-2017\n",
      "Lean and ambitious, unsentimental and bombastic, overwhelmingly guy-centric, Christopher Nolan's World War II epic \"Dunkirk\" showcases the best and worst of the director's tendencies. The best win out and the worst recede in memory when you think back on the experience—provided that you want to remember \"Dunkirk,\" a movie that's supposed to be grueling and succeeds. Less of a war film and more of a disaster (or survival) picture, it's an ensemble work that chronicles the evacuation of British soldiers who got trapped in the harbor and on the beaches of Dunkirk, France, in late May and early June of 1940, with the Germans, who had driven Allied forces practically out to sea, closing in for one last sweep.\n",
      "\n",
      "If you were to make a list of every phobia you can think of, you'd have to tick off a lot of boxes after seeing this film. Fear of heights, fire, drowning, confined spaces, darkness, abandonment—you name it, it's represented in cinematographer Hoyte van Hoytema's nightmarishly clear images. And if you see the movie in one of the handful of theaters showing it in 70mm IMAX format, the experience will feel even more constricting and oppressive because of the image's unusual shape. It's close to the old-fashioned \"Academy\" ratio common to films made in cinema's early decades: squarish, tall instead of wide. That means that when you're in the cockpit of a fighter  diving towards the water, or running behind an infantryman dodging German snipers, the idea of \"tunnel vision,\" a phrase spoken by many a catastrophe survivor, comes to life onscreen. \n",
      "\n",
      "The film will be shown in a wider format in most cinemas, but I doubt this will lessen the overall effect: this is a pile-driver of a movie, dropping one visual or aural bomb after another, with barely a pause to contemplate what it's just shown you. To watch it is to feel beleaguered. This was a period in which German military power was ascendant and hope for the United Kingdom's survival was starting to ebb. The story of Dunkirk has been told on film before, notably in Leslie Norman's same-titled 1958 feature, and there has been no shortage of other films about other battlefield rescues; but this one feels different, mainly because of how it's made. \n",
      "\n",
      "Nolan, who also wrote the film's script, drops you into the middle of the action from frame one and keeps you there. This is an ensemble movie that doesn't just fail to delineate most of its characters through exposition but seems to take perverse pride in letting them scamper anonymously across the screen at flyspeck distance, getting lost amid crowds or merging with smoke or water. Scenes sometimes play out for minutes without audible dialogue, a rarity in commercial cinema made at this budget level; it's even rarer in Nolan's own films, which tend to clarify narrative via massive verbal exposition dumps. Nolan and van Hoytema hold shots longer than the Nolan norm, sometimes long enough to let you consider everything in the frame and decide where to let your eye settle. \n",
      "\n",
      "Like a more restless cousin of Terrence Malick, who infused the combat picture with Transcendental philosophy in \"The Thin Red Line,\" or Robert Altman, who painted microcosmic panoramas of civilization in such films as \"Nashville\" and \"Short Cuts,\" \"Dunkirk\" treats every person on that beach and in assorted nearby planes and boats as part of a collective organism, less interesting for their biographical details than for the roles they play in the drama of history, however large or small they may be. \"Dunkirk\" is what I like to call an Ant Farm Picture: it's a portrait of a society, or a species, fighting for its life. It's not hugely interested in the plight of individuals, unless they're trying to save themselves or others. If you get confused about who's who and what's what from time to time, you can rest assured that this is a feature of Nolan's methods, not a bug (pun intended). \n",
      "\n",
      "Tom Hardy plays a fighter pilot trying to blast German pilots out of the sky before they can strafe soldiers on the ground and sink boats in the harbor. He has maybe a dozen lines and spends much of the film behind a mask, as he did in his last collaboration with Nolan, \"The Dark Knight Rises\"; but he makes a strong impression anyway by treating the character as the sum total of his actions. Mark Rylance plays a civilian with teenage sons who is determined to pilot his small yacht to Dunkirk and rescue as many people as he can; there are lots of these self-appointed rescuers around Dunkirk; their ultimate organization into one of the twentieth century's boldest non-military flotillas is as inspiring as you imagine it to be. A trio of soldiers, one of whom is played by Harry Styles, rushes from the town to the beach and onto a long dock that stretches into the ocean; this is the only way that big boats can get close enough to shore to pick up the stranded. The would-be passengers pray that they can pile onto a ship and get out before more German planes shred them with bullets or bombs. Some of the characters, including Hardy's Farrier and Rylance's Mark Dawson or Kenneth Branagh's Commander Bolton, the highest ranking English officer on the scene, are given names. Others are identified only by their general appearance or actions, such as Cillian Murphy, known only as \"Shivering Soldier\"; he's pulled from the icy sea by Rylance's captain and strongly urges the crew to sail away from Dunkirk, not toward it.\n",
      "\n",
      "The film has its share of stumbling blocks. One is the persistent anonymity of the characters; just because a gambit is a conscious part of the film's design doesn't mean it always works, and there are moments you may wonder whether treating supporting players as something other than glorified cannon fodder might have resulted in a film as emotionally powerful as it is viscerally overwhelming. Another miscalculation is the score, by Hans Zimmer, a Jungian din of booming drums, bum-vibrating synth chords, and cawing string effects that loses much of its power by refusing to shut up, even when silence or ambient war noise might have been just as effective, or more so. The overuse of Zimmer's music has been an issue throughout Nolan's career, but here may become an object of debate. The situations and images are so vivid that the score often seems to be trying to rescue a film that doesn't need its help.\n",
      "\n",
      "I was more on-the-fence about the movie's intricate narrative construction, but once the film's visceral impact had faded, it was there that my mind wandered. Like most of Nolan's films, \"Dunkirk\" is obsessed by the relative perception of time. This is emphasized here by the cross-cutting of Lee Smith. Smith has edited all of Nolan's movies since \"Batman Begins\"—including \"Interstellar,\" which is explicitly about the idea of time passing more quickly or slowly depending on where you are. \"Dunkirk\" tells us in its chapter-like opening titles that one major subplot takes place over a week, another in a day, and yet another in one hour. Then the movie hops between them in ways that compress and expand time for poetic effect—making, say, a plane's run that probably took thirty seconds seem to take exactly as long as a sea rescue that lasted hours. \n",
      "\n",
      "One could make a case that this amounts to over-intellectualization of a strong, simple tale. But that's been Nolan's m.o. from \"Following\" and \"Memento\" onward, and I'd be lying if I said it didn't fascinate me, even if a particular film isn't doing much for me scene-to-scene. It has often been said that trauma wreaks havoc with one's perceptions of time. This is one of the few works I can think of that considers that idea over the course of a whole feature, not just in self-contained sequences. (The backbone of Zimmer's score, appropriately, is a ticking clock.)\n",
      "\n",
      "If somebody were to ask me if I liked this film, I would tell them no. I loathed parts of it and found other parts repetitious or half-baked. But, maybe paradoxically, I admired it throughout, and have been thinking about it constantly since I saw it. Even the aspects of \"Dunkirk\" that didn't sit right with me are all of a piece. This is a movie of vision and integrity made on an epic scale, a series of propositions dramatized with machines, bodies, seawater and fire. It deserves to be seen and argued about. They don't make them like this anymore. Never did, really.\n"
     ]
    }
   ],
   "source": [
    "for single_file in glob.glob('Reviews/*.txt'): #this is the wildcard 'Reviews/*.txt'\n",
    "       with open(single_file, encoding='utf-8') as file: \n",
    "            print(file.read())\n",
    "            break;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I dont need all that !\n",
    "## I need the first 3 pices of data:\n",
    "- Title -first row\n",
    "- URL -2nd row\n",
    "- Review -third paragraph and all remaining\n",
    "\n",
    "So, i'll not use `file.read()` <br>\n",
    "text files are **seperated with new line char**, I will read the file line by line\n",
    "- to read one line, I will use `file.readline()`, lets check this with `print()` and `break;` again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dunkirk (2017)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for single_file in glob.glob('Reviews/*.txt'): #this is the wildcard 'Reviews/*.txt'\n",
    "       with open(single_file, encoding='utf-8') as file: \n",
    "            print(file.readline())\n",
    "            break;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Theh `\\n` or the nwe line char made the white space above** <br>\n",
    "So, i'll slice off at the end of the line/string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dunkirk (2017)\n"
     ]
    }
   ],
   "source": [
    "for single_file in glob.glob('Reviews/*.txt'): #this is the wildcard 'Reviews/*.txt'\n",
    "       with open(single_file, encoding='utf-8') as file: \n",
    "            print(file.readline()[:-1])\n",
    "            break;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Now, lets get the url and the full review text!** <br>\n",
    "I will make a list and then convert it to panda frame.. <br>\n",
    "and then fill the list with dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_list = []\n",
    "for single_review in glob.glob('Reviews/*.txt'):\n",
    "    with open(single_review, encoding='utf-8') as file:\n",
    "        title = file.readline()[:-1] #first line\n",
    "        #print(title)\n",
    "        review_url = file.readline()[:-1] #second line\n",
    "        #print(review_url)\n",
    "        review_text = file.read() #third & everything left (explained next)\n",
    "        #print(review_text)\n",
    "        #break;\n",
    "        # Append to list of dictionaries\n",
    "        df_list.append({'title': title,\n",
    "                        'review_url': review_url,\n",
    "                        'review_text': review_text})\n",
    "df = pd.DataFrame(df_list, columns = ['title', 'review_url', 'review_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "file object returned by open() is not only an iterable, it is an iterator<br>\n",
    "which is why it can only traverse the file once.<br>\n",
    "You may reset the file cursor with .seek(0) as many have suggested <br>\n",
    "but you should, in most cases, only iterate a file once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>review_url</th>\n",
       "      <th>review_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Dunkirk (2017)</td>\n",
       "      <td>http://www.rogerebert.com/reviews/dunkirk-2017</td>\n",
       "      <td>Lean and ambitious, unsentimental and bombasti...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Army of Shadows (L'Armée des ombres) (1969)</td>\n",
       "      <td>http://www.rogerebert.com/reviews/great-movie-...</td>\n",
       "      <td>Jean-Pierre Melville's \"Army of Shadows\" is ab...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Alien (1979)</td>\n",
       "      <td>http://www.rogerebert.com/reviews/great-movie-...</td>\n",
       "      <td>At its most fundamental level, \"Alien\" is a mo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The Bride of Frankenstein (1935)</td>\n",
       "      <td>http://www.rogerebert.com/reviews/great-movie-...</td>\n",
       "      <td>To a new world of gods and monsters.\\n\\nSo int...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>The 400 Blows (Les Quatre cents coups) (1959)</td>\n",
       "      <td>http://www.rogerebert.com/reviews/great-movie-...</td>\n",
       "      <td>I demand that a film express either the joy of...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           title  \\\n",
       "0                                 Dunkirk (2017)   \n",
       "1    Army of Shadows (L'Armée des ombres) (1969)   \n",
       "2                                   Alien (1979)   \n",
       "3               The Bride of Frankenstein (1935)   \n",
       "4  The 400 Blows (Les Quatre cents coups) (1959)   \n",
       "\n",
       "                                          review_url  \\\n",
       "0     http://www.rogerebert.com/reviews/dunkirk-2017   \n",
       "1  http://www.rogerebert.com/reviews/great-movie-...   \n",
       "2  http://www.rogerebert.com/reviews/great-movie-...   \n",
       "3  http://www.rogerebert.com/reviews/great-movie-...   \n",
       "4  http://www.rogerebert.com/reviews/great-movie-...   \n",
       "\n",
       "                                         review_text  \n",
       "0  Lean and ambitious, unsentimental and bombasti...  \n",
       "1  Jean-Pierre Melville's \"Army of Shadows\" is ab...  \n",
       "2  At its most fundamental level, \"Alien\" is a mo...  \n",
       "3  To a new world of gods and monsters.\\n\\nSo int...  \n",
       "4  I demand that a film express either the joy of...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}