commited all contents created by participants
This commit is contained in:
Bachir Soussi Chiadmi
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,10 @@
|
||||
THE YEAR 1866 was marked by a bizarre development, an unexplained and downright inexplicable phenomenon that surely no one has forgotten. Without getting into those rumors that upset civilians in the seaports and deranged the public mind even far inland, it must be said that professional seamen were especially alarmed. Traders, shipowners, captains of vessels, skippers, and master mariners from Europe and America, naval officers from every country, and at their heels the various national governments on these two continents, were all extremely disturbed by the business.
|
||||
In essence, over a period of time several ships had encountered "an enormous thing" at sea, a long spindle-shaped object, sometimes giving off a phosphorescent glow, infinitely bigger and faster than any whale.
|
||||
The relevant data on this apparition, as recorded in various logbooks, agreed pretty closely as to the structure of the object or creature in question, its unprecedented speed of movement, its startling locomotive power, and the unique vitality with which it seemed to be gifted. If it was a cetacean, it exceeded in bulk any whale previously classified by science. No naturalist, neither Cuvier nor Lacépède, neither Professor Dumeril nor Professor de Quatrefages, would have accepted the existence of such a monster sight unseen-- specifically, unseen by their own scientific eyes.
|
||||
Striking an average of observations taken at different times-- rejecting those timid estimates that gave the object a length of 200 feet, and ignoring those exaggerated views that saw it as a mile wide and three long--you could still assert that this phenomenal creature greatly exceeded the dimensions of anything then known to ichthyologists, if it existed at all.
|
||||
Now then, it did exist, this was an undeniable fact; and since the human mind dotes on objects of wonder, you can understand the worldwide excitement caused by this unearthly apparition. As for relegating it to the realm of fiction, that charge had to be dropped.
|
||||
In essence, on July 20, 1866, the steamer Governor Higginson, from the Calcutta & Burnach Steam Navigation Co., encountered this moving mass five miles off the eastern shores of Australia.
|
||||
Captain Baker at first thought he was in the presence of an unknown reef; he was even about to fix its exact position when two waterspouts shot out of this inexplicable object and sprang hissing into the air some 150 feet. So, unless this reef was subject to the intermittent eruptions of a geyser, the Governor Higginson had fair and honest dealings with some aquatic mammal, until then unknown, that could spurt from its blowholes waterspouts mixed with air and steam.
|
||||
Similar events were likewise observed in Pacific seas, on July 23 of the same year, by the Christopher Columbus from the West India & Pacific Steam Navigation Co. Consequently, this extraordinary cetacean could transfer itself from one locality to another with startling swiftness, since within an interval of just three days, the Governor Higginson and the Christopher Columbus had observed it at two positions on the charts separated by a distance of more than 700 nautical leagues.
|
||||
Fifteen days later and 2,000 leagues farther, the Helvetia from the Compagnie Nationale and the Shannon from the Royal Mail line, running on opposite tacks in that part of the Atlantic lying between the United States and Europe, respectively signaled each other that the monster had been sighted in latitude 42 degrees 15' north and longitude 60 degrees 35' west of the meridian of Greenwich. From their simultaneous observations, they were able to estimate the mammal's minimum length at more than 350 English feet; this was because both the Shannon and the Helvetia were of smaller dimensions, although each measured 100 meters stem to stern. Now then, the biggest whales, those rorqual whales that frequent the waterways of the Aleutian Islands, have never exceeded a length of 56 meters--if they reach even that.
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
El año 1866 quedó caracterizado por un extraño acontecimiento, por un fenómeno inexplicable e inexplicado que nadie, sin duda, ha podido olvidar. Sin hablar de los rumores que agitaban a las poblaciones de los puertos y que sobreexcitaban a los habitantes del interior de los continentes, el misterioso fenómeno suscitó una particular emoción entre los hombres del mar. Negociantes, armadores, capitanes de barco, skippers y masters de Europa y de América, oficiales de la marina de guerra de todos los países y, tras ellos, los gobiernos de los diferentes Estados de los dos continentes, manifestaron la mayor preocupación por el hecho.
|
||||
Desde hacía algún tiempo, en efecto, varios barcos se habían encontrado en sus derroteros con «una cosa enorme», con un objeto largo, fusiforme, fosforescente en ocasiones, infinitamente más grande y más rápido que una ballena.
|
||||
Los hechos relativos a estas apariciones, consignados en los diferentes libros de a bordo, coincidían con bastante exactitud en lo referente a la estructura del objeto o del ser en cuestión, a la excepcional velocidad de sus movimientos, a la sorprendente potencia de su locomoción y a la particular vitalidad de que parecía dotado. De tratarse de un cetáceo, superaba en volumen a todos cuantos especímenes de este género había clasificado la ciencia hasta entonces. Ni Cuvier, ni Lacepède, ni Dumeril ni Quatrefages hubieran admitido la existencia de tal monstruo, a menos de haberlo visto por sus propios ojos de sabios
|
||||
El promedio de las observaciones efectuadas en diferentes circunstancias una vez descartadas tanto las tímidas evaluaciones que asignaban a ese objeto una longitud de doscientos pies, como las muy exageradas que le imputaban una anchura de una milla y una longitud de tres permitía afirmar que ese ser fenomenal, de ser cierta su existencia, superaba con exceso todas las dimensiones admitidas hasta entonces por los ictiólogos.
|
||||
Pero existía; innegable era ya el hecho en sí mismo. Y, dada esa inclinación a lo maravilloso que existe en el hombre, se comprende la emoción producida por esa sobrenatural aparición. Preciso era renunciar a la tentación de remitirla al reino de las fábulas.
|
||||
Efectivamente, el 20 de julio de 1866, el vapor Governor Higginson, de la Calcuta and Burnach Steam Navigation Company, había encontrado esa masa móvil a cinco millas al este de las costas de Australia. El capitán Baker creyó, al pronto, hallarse en presencia de un escollo desconocido, y se disponía a determinar su exacta situación cuando pudo ver dos columnas de agua, proyectadas por el inexplicable objeto, elevarse silbando por el aire hasta ciento cincuenta pies. Forzoso era, pues, concluir que de no estar el escollo sometido a las expansiones intermitentes de un géiser, el Governor Higginson había encontrado un mamífero acuático, desconocido hasta entonces, que expulsaba por sus espiráculos columnas de agua, mezcladas con aire y vapor.
|
||||
Se observó igualmente tal hecho el 23 de julio del mismo año, en aguas del Pacífico, por el Cristóbal Colón, de la West India and Pacific Steam Navigation Company,. Por consiguiente, el extraordinario cetáceo podía trasladarse de un lugar a otro con una velocidad sorprendente, puesto que, a tres días de intervalo tan sólo, el Governor Higginson y el Cristóbal Colón lo habían observado en dos puntos del mapa separados por una distancia de más de setecientas leguas marítimas.
|
||||
Quince días más tarde, a dos mil leguas de allí, el Helvetia, de la Compagnie Nationale, y el Shannon, de la Royal Mail, navegando en sentido opuesto por la zona del Atlántico comprendida entre Europa y Estados Unidos, se señalaron mutuamente al monstruo a 42º 15' de latitud norte y 60º 35' de longitud al oeste del meridiano de Greenwich. En esa observación simultánea se creyó poder evaluar la longitud mínima del mamífero en más de trescientos cincuenta pies ingleses, dado que el Shannon y el Helvetia eran de dimensiones inferiores, aun cuando ambos midieran cien metros del tajamar al codaste. Ahora bien, las ballenas más grandes, las que frecuentan los parajes de las islas Aleutinas, la Kulammak y la Umgullick, no sobrepasan los cincuenta y seis metros de longitud, si es que llegan a alcanzar tal dimensión.
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,8 @@
|
||||
L'année 1866 fut marquée par un événement bizarre, un phénomène inexpliqué et inexplicable que personne n'a sans doute oublié. Sans parler des rumeurs qui agitaient les populations des ports et surexcitaient l'esprit public à l'intérieur des continents les gens de mer furent particulièrement émus. Les négociants, armateurs, capitaines de navires, skippers et masters de l'Europe et de l'Amérique, officiers des marines militaires de tous pays, et, après eux, les gouvernements des divers États des deux continents, se préoccupèrent de ce fait au plus haut point.
|
||||
En effet, depuis quelque temps, plusieurs navires s'étaient rencontrés sur mer avec «une chose énorme» un objet long, fusiforme, parfois phosphorescent, infiniment plus vaste et plus rapide qu'une baleine.
|
||||
Les faits relatifs à cette apparition, consignés aux divers livres de bord, s'accordaient assez exactement sur la structure de l'objet ou de l'être en question, la vitesse inouïe de ses mouvements, la puissance surprenante de sa locomotion, la vie particulière dont il semblait doué. Si c'était un cétacé, il surpassait en volume tous ceux que la science avait classés jusqu'alors. Ni Cuvier, ni Lacépède, ni M. Dumeril, ni M. de Quatrefages n'eussent admis l'existence d'un tel monstre — à moins de l'avoir vu, ce qui s'appelle vu de leurs propres yeux de savants.
|
||||
A prendre la moyenne des observations faites à diverses reprises — en rejetant les évaluations timides qui assignaient à cet objet une longueur de deux cents pieds et en repoussant les opinions exagérées qui le disaient large d'un mille et long de trois — on pouvait affirmer, cependant, que cet être phénoménal dépassait de beaucoup toutes les dimensions admises jusqu'à ce jour par les ichtyologistes — s'il existait toutefois.
|
||||
Or, il existait, le fait en lui-même n'était plus niable, et, avec ce penchant qui pousse au merveilleux la cervelle humaine, on comprendra l'émotion produite dans le monde entier par cette surnaturelle apparition. Quant à la rejeter au rang des fables, il fallait y renoncer.
|
||||
En effet, le 20 juillet 1866, le steamer Governor-Higginson, de Calcutta and Burnach steam navigation Company, avait rencontré cette masse mouvante à cinq milles dans l'est des côtes de l'Australie. Le capitaine Baker se crut, tout d'abord, en présence d'un écueil inconnu; il se disposait même à en déterminer la situation exacte, quand deux colonnes d'eau, projetées par l'inexplicable objet, s'élancèrent en sifflant à cent cinquante pieds dans l'air. Donc, à moins que cet écueil ne fût soumis aux expansions intermittentes d'un geyser, le Governor-Higginson avait affaire bel et bien à quelque mammifère aquatique, inconnu jusque-là, qui rejetait par ses évents des colonnes d'eau, mélangées d'air et de vapeur
|
||||
Pareil fait fut également observé le 23 juillet de la même année, dans les mers du Pacifique, par le Cristobal-Colon, de West India and Pacific steam navigation Company. Donc, ce cétacé extraordinaire pouvait se transporter d'un endroit à un autre avec une vélocité surprenante, puisque à trois jours d'intervalle, le Governor-Higginson et le Cristobal-Colon l'avaient observé en deux points de la carte séparés par une distance de plus de sept cents lieues marines. Quinze jours plus tard, à deux mille lieues de là l'Helvetia, de la Compagnie Nationale, et le Shannon, du Royal-Mail, marchant à contrebord dans cette portion de l'Atlantique comprise entre les États-Unis et l'Europe, se signalèrent respectivement le monstre par 42°15' de latitude nord, et 60°35' de longitude à l'ouest du méridien de Greenwich. Dans cette observation simultanée, on crut pouvoir évaluer la longueur minimum du mammifère à plus de trois cent cinquante pieds anglais, puisque le Shannon et l'Helvetia étaient de dimension inférieure à lui, bien qu'ils mesurassent cent mètres de l'étrave à l'étambot. Or, les plus vastes baleines, celles qui fréquentent les parages des îles Aléoutiennes, le Kulammak et l'Umgullick, n'ont jamais dépassé la longueur de cinquante-six mètres, — si même elles l'atteignent.
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
L'année 1866 fut marquée par un événement bizarre, un phénomène inexpliqué et inexplicable que personne n'a sans doute oublié. Sans parler des rumeurs qui agitaient les populations des ports et surexcitaient l'esprit public à l'intérieur des continents les gens de mer furent particulièrement émus. Les négociants, armateurs, capitaines de navires, skippers et masters de l'Europe et de l'Amérique, officiers des marines militaires de tous pays, et, après eux, les gouvernements des divers États des deux continents, se préoccupèrent de ce fait au plus haut point.
|
||||
En effet, depuis quelque temps, plusieurs navires s'étaient rencontrés sur mer avec «une chose énorme» un objet long, fusiforme, parfois phosphorescent, infiniment plus vaste et plus rapide qu'une baleine.
|
||||
Les faits relatifs à cette apparition, consignés aux divers livres de bord, s'accordaient assez exactement sur la structure de l'objet ou de l'être en question, la vitesse inouïe de ses mouvements, la puissance surprenante de sa locomotion, la vie particulière dont il semblait doué. Si c'était un cétacé, il surpassait en volume tous ceux que la science avait classés jusqu'alors. Ni Cuvier, ni Lacépède, ni M. Dumeril, ni M. de Quatrefages n'eussent admis l'existence d'un tel monstre — à moins de l'avoir vu, ce qui s'appelle vu de leurs propres yeux de savants.
|
||||
A prendre la moyenne des observations faites à diverses reprises — en rejetant les évaluations timides qui assignaient à cet objet une longueur de deux cents pieds et en repoussant les opinions exagérées qui le disaient large d'un mille et long de trois — on pouvait affirmer, cependant, que cet être phénoménal dépassait de beaucoup toutes les dimensions admises jusqu'à ce jour par les ichtyologistes — s'il existait toutefois.
|
||||
Or, il existait, le fait en lui-même n'était plus niable, et, avec ce penchant qui pousse au merveilleux la cervelle humaine, on comprendra l'émotion produite dans le monde entier par cette surnaturelle apparition. Quant à la rejeter au rang des fables, il fallait y renoncer.
|
||||
En effet, le 20 juillet 1866, le steamer Governor-Higginson, de Calcutta and Burnach steam navigation Company, avait rencontré cette masse mouvante à cinq milles dans l'est des côtes de l'Australie. Le capitaine Baker se crut, tout d'abord, en présence d'un écueil inconnu; il se disposait même à en déterminer la situation exacte, quand deux colonnes d'eau, projetées par l'inexplicable objet, s'élancèrent en sifflant à cent cinquante pieds dans l'air. Donc, à moins que cet écueil ne fût soumis aux expansions intermittentes d'un geyser, le Governor-Higginson avait affaire bel et bien à quelque mammifère aquatique, inconnu jusque-là, qui rejetait par ses évents des colonnes d'eau, mélangées d'air et de vapeur
|
||||
Pareil fait fut également observé le 23 juillet de la même année, dans les mers du Pacifique, par le Cristobal-Colon, de West India and Pacific steam navigation Company. Donc, ce cétacé extraordinaire pouvait se transporter d'un endroit à un autre avec une vélocité surprenante, puisque à trois jours d'intervalle, le Governor-Higginson et le Cristobal-Colon l'avaient observé en deux points de la carte séparés par une distance de plus de sept cents lieues marines. Quinze jours plus tard, à deux mille lieues de là l'Helvetia, de la Compagnie Nationale, et le Shannon, du Royal-Mail, marchant à contrebord dans cette portion de l'Atlantique comprise entre les États-Unis et l'Europe, se signalèrent respectivement le monstre par 42°15' de latitude nord, et 60°35' de longitude à l'ouest du méridien de Greenwich. Dans cette observation simultanée, on crut pouvoir évaluer la longueur minimum du mammifère à plus de trois cent cinquante pieds anglais, puisque le Shannon et l'Helvetia étaient de dimension inférieure à lui, bien qu'ils mesurassent cent mètres de l'étrave à l'étambot. Or, les plus vastes baleines, celles qui fréquentent les parages des îles Aléoutiennes, le Kulammak et l'Umgullick, n'ont jamais dépassé la longueur de cinquante-six mètres, — si même elles l'atteignent.
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,4 @@
|
||||
let1 =o"
|
||||
let2 = "l"
|
||||
let3 ="a"
|
||||
print(let1+let2+let3)
|
||||
@@ -0,0 +1,4 @@
|
||||
let1 ="o"
|
||||
let2 = "l"
|
||||
let3 ="a"
|
||||
print(let1+let2+let3)
|
||||
+304
@@ -0,0 +1,304 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Building a simple word count application with Spark\n",
|
||||
"\n",
|
||||
"This lab will build on the techniques covered in the first Spark workshop. We will develop a simple word count application of the most common words in the [Complete Works of William Shakespeare](http://www.gutenberg.org/ebooks/100) retrieved from [Project Gutenberg](http://www.gutenberg.org/wiki/Main_Page). \n",
|
||||
"\n",
|
||||
"This lab is mandatory for Workshop 2 and required to validate your registration. \n",
|
||||
"\n",
|
||||
"####Read-me before:\n",
|
||||
"You must execute each cell and fill with the appropriate code when necessary.\n",
|
||||
"At the end of the notebook, there is a generated code to be copied and pasted into the meetup registration. "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Setup import and functions"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Just excecute this cell\n",
|
||||
"import os.path\n",
|
||||
"import re\n",
|
||||
"import hashlib"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Loads the [Complete Works of William Shakespeare](http://www.gutenberg.org/ebooks/100) retrieved from [Project Gutenberg](http://www.gutenberg.org/wiki/Main_Page)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Just excecute this cell\n",
|
||||
"baseDir = os.path.join('data')\n",
|
||||
"inputPath = os.path.join('shakespeare.txt')\n",
|
||||
"fileName = os.path.join(baseDir, inputPath)\n",
|
||||
"\n",
|
||||
"shakespeareRDD = (sc\n",
|
||||
" .textFile(fileName, 8))\n",
|
||||
"\n",
|
||||
"shakespeareRDD.cache()\n",
|
||||
"print '\\n'.join(shakespeareRDD\n",
|
||||
" .zipWithIndex() # to (line, lineNum)\n",
|
||||
" .map(lambda (l, num): '{0}: {1}'.format(num, l)) # to 'lineNum: line'\n",
|
||||
" .take(15))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Just excecute this cell\n",
|
||||
"def toLower(text):\n",
|
||||
" \"\"\"\n",
|
||||
" Changes all text to lower case.\n",
|
||||
" \"\"\"\n",
|
||||
" return text.lower()\n",
|
||||
"\n",
|
||||
"print toLower('Hello WORLD') #should be \"hello world\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Define the function `removePunctuation` removes any punctuation. We use the Python [re](https://docs.python.org/2/library/re.html) module to remove any text that is not a letter, number, or space."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Just excecute this cell\n",
|
||||
"pattern=re.compile(\"[^a-zA-Z0-9\\s]\")\n",
|
||||
"def removePunctuation(text):\n",
|
||||
" \"\"\"Removes punctuation from the given text\n",
|
||||
"\n",
|
||||
" Note:\n",
|
||||
" Only spaces, letters, and numbers should be retained. Other characters should should be\n",
|
||||
" eliminated (e.g. it's becomes its). Leading and trailing spaces should be removed after\n",
|
||||
" punctuation is removed.\n",
|
||||
"\n",
|
||||
" Args:\n",
|
||||
" text (str): A string.\n",
|
||||
"\n",
|
||||
" Returns:\n",
|
||||
" str: The cleaned up string.\n",
|
||||
" \"\"\"\n",
|
||||
" cleanText = pattern.sub('', text)\n",
|
||||
" return cleanText\n",
|
||||
"print removePunctuation('Hi, you! My ZIP code is 98-9800') #should be Hi you My ZIP code is 989800\n",
|
||||
"print removePunctuation('No under_score!') #No underscore"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Just excecute this cell\n",
|
||||
"def strips(text):\n",
|
||||
" \"\"\"strips leading and trailing spaces.\n",
|
||||
" \"\"\"\n",
|
||||
" return text.strip()\n",
|
||||
"print '>%s<' % strips(' This is a text') #should print >This is a text<\n",
|
||||
"print '>%s<' % (strips(removePunctuation('No under_score !'))) #should print >No underscore<"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Just excecute this cell\n",
|
||||
"stopfile = os.path.join(baseDir, 'stopwords.txt')\n",
|
||||
"stopwords = set(sc.textFile(stopfile).collect())\n",
|
||||
"print 'These are the stopwords: %s' % stopwords"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Just excecute this cell\n",
|
||||
"def isNotStopWord(word):\n",
|
||||
" \"\"\" Tells if the given word isn't a English common word.\n",
|
||||
" Args:\n",
|
||||
" string (str): input string\n",
|
||||
" Returns:\n",
|
||||
" Boolean: True if word isn't a stopword. Otherwise, False\n",
|
||||
" \"\"\"\n",
|
||||
" return word not in stopwords\n",
|
||||
"\n",
|
||||
"print isNotStopWord('brown') # Should give True\n",
|
||||
"print isNotStopWord('the') # Should give False"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### wordCount` function **\n",
|
||||
"#### First, define a function for word counting. You should reuse the techniques that have been covered during the first workshop. This function should take in an RDD that is a list of words and return a pair RDD that has all of the words and their associated counts."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO: Replace <FILL IN> with appropriate code\n",
|
||||
"def wordCount(wordListRDD):\n",
|
||||
" \"\"\"Creates a pair RDD with word counts from an RDD of words.\n",
|
||||
" Args:\n",
|
||||
" wordListRDD (RDD of str): An RDD consisting of words.\n",
|
||||
"\n",
|
||||
" Returns:\n",
|
||||
" RDD of (str, int): An RDD consisting of (word, count) tuples.\n",
|
||||
" \"\"\"\n",
|
||||
" return <FILL IN>"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Before you can use the `wordcount()` function, you have to address two issues with the format of the RDD:\n",
|
||||
" + #### The first issue is that that we need to split each line by its spaces.\n",
|
||||
" + #### The second issue is we need to filter out empty lines.\n",
|
||||
" \n",
|
||||
"#### Apply a transformation that will split each element of the RDD by its spaces. You might think that a `map()` transformation is the way to do this, but think about what the result of the `split()` function will be."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# TODO: Replace <FILL IN> with appropriate code\n",
|
||||
"cleanRDD = (shakespeareRDD\n",
|
||||
" .map(removePunctuation)\n",
|
||||
" .map(toLower)\n",
|
||||
" .map(strips)\n",
|
||||
" .<FILL IN>(lambda line: line.split(' '))\n",
|
||||
" .filter(<FILL IN>)\n",
|
||||
" .filter(isNotStopWord))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### You now have an RDD that is only words. Next, let's apply the `wordCount()` function to produce a list of word counts. We can view the top 15 words by using the `takeOrdered()` action; however, since the elements of the RDD are pairs, we need a custom sort function that sorts using the value part of the pair.\n",
|
||||
"\n",
|
||||
"#### Use the `wordCount()` function and `takeOrdered()` to obtain the fifteen most common words and their counts."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#collect the top 15\n",
|
||||
"top15WordsAndCounts = wordCount(cleanRDD).<FILL IN>\n",
|
||||
"print '\\n'.join(map(lambda (w, c): '{0}: {1}'.format(w, c), top15WordsAndCounts))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"####Generate the md5 code to validate your registration"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"md5_code = hashlib.md5()\n",
|
||||
"for (word, count) in top15WordsAndCounts:\n",
|
||||
" md5_code.update(word)\n",
|
||||
"\n",
|
||||
"meetup_code = md5_code.hexdigest()\n",
|
||||
"if hashlib.sha224(meetup_code).hexdigest() == '427681d5929a35ab878c291b0de5f4b8a009dc9b71d2e54dbf7c46ba':\n",
|
||||
" print 'Well done, copy this code: %s' % md5_code.hexdigest()\n",
|
||||
"else:\n",
|
||||
" print 'This is not the expected code, please try again. \\nTip: the code starts with \"cc\" and finishes with \"ad1c\"'"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 2",
|
||||
"language": "python",
|
||||
"name": "python2"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 2
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython2",
|
||||
"version": "2.7.13"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 0
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
let1 ="o"
|
||||
let2 = "l"
|
||||
let3 ="a"
|
||||
print(let1+let2+let3)
|
||||
+45
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright (C) 2017 Constant, Algolit
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details: <http://www.gnu.org/licenses/>.
|
||||
|
||||
|
||||
'''
|
||||
Input texts are checked against a dictionary that assigns weights to different vowels.
|
||||
The script gives a score for a specific sentence.
|
||||
'''
|
||||
|
||||
# create a dictionary
|
||||
scrabble = {'a': 3, 'e': 1, 'i': 2, 'o': 4, 'u':4, 'y': 6}
|
||||
|
||||
# set weight to 0
|
||||
weights = 0
|
||||
|
||||
# find a sentence / string
|
||||
sentence = "La vie est un mystère qu'il faut vivre, et non un problème à résoudre."
|
||||
|
||||
# spllit sentence in list of words
|
||||
words = sentence.split()
|
||||
|
||||
# for each word
|
||||
for word in words:
|
||||
# iterate over letters of word
|
||||
for letter in word:
|
||||
# check if letter is in dictionary
|
||||
if letter in scrabble:
|
||||
# if yes, get weight of the letter
|
||||
weight = scrabble[letter]
|
||||
# add letter weight to general weight
|
||||
weights += weight
|
||||
|
||||
# print general weight
|
||||
print("Le poids de ma phrase est:", weights)
|
||||
@@ -0,0 +1,12 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
with open("JVFR.txt", "r") as textFr:
|
||||
for line in textFr:
|
||||
words=line.split
|
||||
for word in words :
|
||||
print(word)
|
||||
#nbs=len(words)
|
||||
#print (nbs)
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
with open("JVFR.txt", "r") as textFr:
|
||||
for line in textFr:
|
||||
words=line.split
|
||||
for word in words :
|
||||
print(word)
|
||||
close(JVFR)
|
||||
#nbs=len(words)
|
||||
#print (nbs)
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
with open("JVFR.txt", "r") as textFr:
|
||||
for line in textFr:
|
||||
words=line.split
|
||||
for word in words :
|
||||
print(word)
|
||||
close("JVFR")
|
||||
#nbs=len(words)
|
||||
#print (nbs)
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
with open("JVFR.txt", "r") as textFr:
|
||||
for line in textFr:
|
||||
line=line.strip
|
||||
words=line.split
|
||||
for word in words :
|
||||
print(word)
|
||||
close("JVFR")
|
||||
#nbs=len(words)
|
||||
#print (nbs)
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
with open("JVFR.txt", "r") as textFr:
|
||||
for line in textFr:
|
||||
line=line.strip()
|
||||
words=line.split()
|
||||
for word in words :
|
||||
print(word)
|
||||
close("JVFR")
|
||||
#nbs=len(words)
|
||||
#print (nbs)
|
||||
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
textFr ="j'ai fait le sport : le jogging"
|
||||
textEn = "I do sport : jogging "
|
||||
mots=textFr.split()
|
||||
words=textEn.split()
|
||||
# for each word in the line:
|
||||
#for mot in mots:
|
||||
# print the "mot"
|
||||
#print(mot)
|
||||
#for word in words:
|
||||
# print the "word"
|
||||
#print(word)
|
||||
for word in words:
|
||||
for mot in mots:
|
||||
if word == mot :
|
||||
print(word)
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env python
|
||||
#-*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
textFr ="L'année 1866 fut marquée par un événement bizarre, un phénomène inexpliqué et inexplicable que personne n'a sans doute oublié. Sans parler des rumeurs qui agitaient les populations des ports et surexcitaient l'esprit public à l'intérieur des continents les gens de mer furent particulièrement émus. Les négociants, armateurs, capitaines de navires, skippers et masters de l'Europe et de l'Amérique, officiers des marines militaires de tous pays, et, après eux, les gouvernements des divers États des deux continents, se préoccupèrent de ce fait au plus haut point. "
|
||||
textEn = "THE YEAR 1866 was marked by a bizarre development, an unexplained and downright inexplicable phenomenon that surely no one has forgotten. Without getting into those rumors that upset civilians in the seaports and deranged the public mind even far inland, it must be said that professional seamen were especially alarmed. Traders, shipowners, captains of vessels, skippers, and master mariners from Europe and America, naval officers from every country, and at their heels the various national governments on these two continents, were all extremely disturbed by the business. "
|
||||
mots=textFr.split()
|
||||
words=textEn.split()
|
||||
# for each word in the line:
|
||||
#for mot in mots:
|
||||
# print the "mot"
|
||||
#print(mot)
|
||||
#for word in words:
|
||||
# print the "word"
|
||||
#print(word)
|
||||
nb = 0
|
||||
for word in words:
|
||||
for mot in mots:
|
||||
if word == mot :
|
||||
nb=nb+1
|
||||
print(word)
|
||||
print("mots commons =" , nb)
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
#!/usr/bin/env python
|
||||
#-*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
textFr ="L'année 1866 fut marquée par un événement bizarre, un phénomène inexpliqué et inexplicable que personne n'a sans doute oublié. Sans parler des rumeurs qui agitaient les populations des ports et surexcitaient l'esprit public à l'intérieur des continents les gens de mer furent particulièrement émus. Les négociants, armateurs, capitaines de navires, skippers et masters de l'Europe et de l'Amérique, officiers des marines militaires de tous pays, et, après eux, les gouvernements des divers États des deux continents, se préoccupèrent de ce fait au plus haut point. "
|
||||
textEn = "THE YEAR 1866 was marked by a bizarre development, an unexplained and downright inexplicable phenomenon that surely no one has forgotten. Without getting into those rumors that upset civilians in the seaports and deranged the public mind even far inland, it must be said that professional seamen were especially alarmed. Traders, shipowners, captains of vessels, skippers, and master mariners from Europe and America, naval officers from every country, and at their heels the various national governments on these two continents, were all extremely disturbed by the business. "
|
||||
mots=textFr.split()
|
||||
words=textEn.split()
|
||||
# for each word in the line:
|
||||
#for mot in mots:
|
||||
# print the "mot"
|
||||
#print(mot)
|
||||
#for word in words:
|
||||
# print the "word"
|
||||
#print(word)
|
||||
nb = 0
|
||||
for word in words:
|
||||
for mot in mots:
|
||||
if word == mot :
|
||||
nb=nb+1
|
||||
print(word)
|
||||
print("mots identiques =" , nb)
|
||||
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env python
|
||||
#-*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
textFr ="L'année 1866 fut marquée par un événement bizarre, un phénomène inexpliqué et inexplicable que personne n'a sans doute oublié. Sans parler des rumeurs qui agitaient les populations des ports et surexcitaient l'esprit public à l'intérieur des continents les gens de mer furent particulièrement émus. Les négociants, armateurs, capitaines de navires, skippers et masters de l'Europe et de l'Amérique, officiers des marines militaires de tous pays, et, après eux, les gouvernements des divers États des deux continents, se préoccupèrent de ce fait au plus haut point. "
|
||||
textEn = "THE YEAR 1866 was marked by a bizarre development, an unexplained and downright inexplicable phenomenon that surely no one has forgotten. Without getting into those rumors that upset civilians in the seaports and deranged the public mind even far inland, it must be said that professional seamen were especially alarmed. Traders, shipowners, captains of vessels, skippers, and master mariners from Europe and America, naval officers from every country, and at their heels the various national governments on these two continents, were all extremely disturbed by the business. "
|
||||
# Convert sentence in list of words
|
||||
mots=textFr.split()
|
||||
# Convert sentence in list of words
|
||||
words=textEn.split()
|
||||
#set nb to zero
|
||||
nb = 0
|
||||
for word in words:
|
||||
for mot in mots:
|
||||
if word == mot :
|
||||
nb=nb+1
|
||||
print(word)
|
||||
print("nombres des mots identiques =" , nb)
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
with open("JVFR.txt", "r") as textFr:
|
||||
for line in textFr:
|
||||
line=line.strip()
|
||||
words=line.split()
|
||||
for word in words :
|
||||
print(word)
|
||||
#close("JVFR")
|
||||
#nbs=len(words)
|
||||
#print (nbs)
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env python
|
||||
# -*- coding: utf-8 -*-
|
||||
# this programme compare a text in french and its translation in englishe and liste the common words .
|
||||
|
||||
with open("JVFR.txt", "r") as textFr:
|
||||
for line in textFr:
|
||||
line=line.strip()
|
||||
mots=line.split()
|
||||
for mot in mots :
|
||||
print(mot)
|
||||
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Reference in New Issue
Block a user