[Migrated project from ks-download Dino Morelli **20150712170108 Ignore-this: 11f7d44ecf1d8c51a9eddb08ecb8d82 Removed and changed everything not related to the common ks-library code. ] move ./ks-download.cabal ./ks-library.cabal hunk ./util/resources/nc-wake/searchResultPage.html 1 - - - -Wake County Restaurant and Sanitation Inspection Grades - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - -
Wake County Restaurant and Sanitation Inspection Grades
-
- -
  -
- - - - - - - - -
- - - - - - - -
- -
-
- - - - - - - -
- -
-
-
- - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - -
User
Pass - - - -
-
- New Account :: - Lost Password? -
-
-
-

Reports

-

For more information on a specific report, or to see the official Health Inspection Report, click on the Facility Heading to expand the list of inspections.

If you're looking for a specific establishment, try the Advanced Report Search function on the right menu, or enter the establishment's name in the search box below.



-
-
- - - - - - - - - - - - - - - - - - - -Jump to facility starting with: - - -
-
- - - - - - - - - - - - - - -
#1 CHINA
- - - - - -
- Location: 764 W Williams ST APEX, NC 27502
- Facility Type: Restaurant
- Inspection Date: 10/28/2014
-
-   -
-
- Score: 96 -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# CommentsPoints
4.2-401.11; Core; Employee beverages may not be stored on or above food or food contact surfaces. Employees still storing foods/beverages on shelves above prep sink. Also had beverages on prep tables. Provide a designated area for employee foods and beverages. 2.0
13.3-302.11; Priority; Store foods in the walk in cooler according to chart provided. Raw shrimp above ready to eat and raw chicken above raw shrimp at the time of inspection. 1.5
47.4-602.13; Core;Cleaning needed on dry storage shelving and outsides of dry storage containers. 0.5
53.6-501.11; Core; Readhere loose stainless steel on wall behind wok. 0.0
54.6-303.11; Core; Replace burned out bulb in hood. 0.0
General Comments
- Red Denotes Critical Violation -
- View Last 3 Inspections for this Establishment -
-
- - - - - - - - - - - - - - -
115 Midtowne
- - - - - -
- Location: 4421-115 Six Forks RD RALEIGH, NC 27609
- Facility Type: Restaurant
- Inspection Date: 09/30/2014
-
-   -
-
- Score: 92 -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# CommentsPoints
2.2-103.11 (M); Priority Foundation; Person in charge unable to provide documentation of an employee health policy that meets the requirements of the Rules. FOOD EMPLOYEES and CONDITIONAL EMPLOYEES are informed of their responsibility to report in accordance with LAW, to the PERSON IN CHARGE, information about their health and activities as they relate to diseases that are transmissible through FOOD, as specified under ¶ 2 201.11(A) CDI-Employee Health Policy Form 1-B was given to management. 1.5
12.3-402.12; No documentation that aqua-cultured salmon was feed pellet or formulated parasite free feed. (C) If raw, raw-marinated, partially cooked, or marinated-partially cooked FISH are served or sold in READY-TO-EAT form, and the FISH are raised and fed as specified in Subparagraph 3 402.11(B)(3), a written agreement or statement from the supplier or aqua-culturist stipulating that the FISH were raised and fed as specified in Subparagraph 3 402.11(B)(3) shall be obtained by the PERSON IN CHARGE and retained in the records of the FOOD ESTABLISHMENT for 90 calendar days beyond the time of service or sale of the FISH. CDI-Education 1.0
14.4-501.114; Priority; Sanitizer shall be 200ppm to 400ppm quat sanitizer when tested with test kits. CDI-fresh sanitizer made. 1.5
20.3-501.16 (A)(2) and (B); Priority;Maintain temperature of chicken, oysters, Pork belly and duck at 45F or below. Temperatures were 48F - 59F. CDI-Food was placed on shallow pan and put in walk-in cooler 1.5
26.7-102.11; Priority Foundation; Working containers used for storing POISONOUS OR TOXIC MATERIALS such as cleaners and SANITIZERS taken from bulk supplies shall be clearly and individually identified with the common name of the material. CDI-Bottle of cleaner labeled 1.0
27.3-502.12; Must follow the below instructions if using ROP without a variance. Do not use reduced oxygen packaging until approved by this department. -(A) Except for a FOOD ESTABLISHMENT that obtains a VARIANCE as specified under § 3502.11, a FOOD ESTABLISHMENT that PACKAGES POTENTIALLY HAZARDOUS FOOD (TIME/TEMPERATURE CONTROL FOR SAFETY FOOD) using a REDUCED OXYGEN PACKAGING method shall control the growth and toxin formation of Clostridium botulinum and the growth of Listeria monocytogenes. P -(B) A FOOD ESTABLISHMENT that PACKAGES POTENTIALLY HAZARDOUS FOOD (TIME/TEMPERATURE CONTROL FOR SAFETY FOOD) using a REDUCED OXYGEN PACKAGING method shall have a HACCP PLAN that contains the information specified under ¶ 8 201.14(D) and that: Pf -(1) Identifies the FOOD to be PACKAGED; Pf -(2) Except as specified under ¶¶ (C) - (E) of this section, requires that the PACKAGED FOOD shall be maintained at 5°C (41°F) or less and meet at least one of the following criteria: Pf -(a) Has an AW of 0.91 or less, Pf -(b) Has a PH of 4.6 or less, Pf -(c) Is a MEAT or POULTRY product cured at a FOOD PROCESSING PLANT regulated by the USDA using substances specified in 9 CFR 424.21, Use of food -ingredients and sources of radiation, and is received in an intact PACKAGE, Pf or -(d) Is a FOOD with a high level of competing organisms such as raw MEAT, raw POULTRY, or raw vegetables; Pf -(3) Describes how the PACKAGE shall be prominently and conspicuously labeled on the principal display panel in bold type on a contrasting background, with instructions to: Pf -(a) Maintain the FOOD at 5oC (41oF) or below, Pf and -(b) Discard the FOOD if within 14 calendar days of its PACKAGING it is not served for on-PREMISES consumption, or consumed if served or sold for off-PREMISES consumption; Pf -(4) Limits the refrigerated shelf life to no more than 14 calendar days from PACKAGING to consumption, except the time the product is maintained frozen, or the original manufacturer’s “sell by” or “use by” date, whichever occurs first; P -(5) Includes operational procedures that: -(a) Prohibit contacting READY-TO-EAT FOOD with bare hands as specified under ¶ 3-301.11(B), Pf -(b) Identify a designated work area and the method by which: Pf -(i) Physical barriers or methods of separation of raw FOODS and READY-TO-EAT FOODS minimize cross contamination, Pf and -(ii) Access to the processing EQUIPMENT is limited to responsible trained personnel familiar with the potential HAZARDS of the operation, Pf and -(c) Delineate cleaning and SANITIZATION procedures for FOOD-CONTACT SURFACES; Pf and -(6) Describes the training program that ensures that the individual responsible for the REDUCED OXYGEN PACKAGING operation understands the: Pf -(a) Concepts required for a safe operation, Pf -(b) EQUIPMENT and facilities, Pf and -(c) Procedures specified under Subparagraph (B)(5) of this section and ¶ 8 201.14(D). Pf -(C) Except for FISH that is frozen before, during, and after PACKAGING, a FOOD ESTABLISHMENT may not PACKAGE FISH using a REDUCED OXYGEN PACKAGING method. P " -(D) Except as specified under ¶ (C) of this section, a FOOD ESTABLISHMENT that PACKAGES FOOD using a cook-chill or sous vide process shall: -(1) Implement a HACCP PLAN that contains the information as specified under ¶ 8-201.14(D); Pf -(2) Ensure the FOOD is: -(a) Prepared and consumed on the PREMISES, or prepared and consumed off the PREMISES but within the same business entity with no distribution or sale of the PACKAGED product to another business entity or the CONSUMER, Pf -(b) Cooked to heat all parts of the FOOD to a temperature and for a time as specified under § 3-401.11, P -(c) Protected from contamination before and after cooking as specified under Parts 3-3 and 3-4, P -(d) Placed in a PACKAGE with an oxygen barrier and sealed before cooking, or placed in a PACKAGE and sealed immediately after cooking and before reaching a temperature below 57°C (135°F), P -(e) Cooled to 5°C (41°F) in the sealed PACKAGE or bag as specified under § 3-501.14 and subsequently: P -(i) Cooled to 1°C (34°F) within 48 hours of reaching 5°C (41°F) and held at that temperature until consumed or discarded within 30 days after the date of PACKAGING;P -(ii) Cooled to 1°C (34°F) within 48 hours of reaching 5°C (41°F), removed from refrigeration equipment that maintains a 1°C (34°F) food temperature and then held at 5°C (41°F) or less for no more than 72 hours, at which time the FOOD must be consumed or discarded; P -(iii) Cooled to 3°C (38°F) or less within 24 hours of reaching 5°C (41°F) and held there for no more than 72 hours from PACKAGING, at which time the food must be consumed or discarded; P or -(iv) Held frozen with no shelf life restriction while -frozen until consumed or used. P -(f) Held in a refrigeration unit that is equipped with an electronic system that continuously monitors time and temperature and is visually examined for proper operation twice daily, Pf -(g) If transported off-site to a satellite location of the same business entity, equipped with verifiable electronic monitoring devices to ensure that times and temperatures are monitored during transportation, Pf and -(h) Labeled with the product name and the date PACKAGED;Pf and -(3) Maintain the records required to confirm that cooling and cold holding refrigeration time/temperature parameters are required as part of the HACCP PLAN and: -(a) Make such records available to the REGULATORY AUTHORITY upon request, Pf and -(b) Hold such records for at least 6 months; Pf and -(4) Implement written operational procedures as specified under Subparagraph (B)(5) of this section and a training program as specified under Subparagraph (B)(6) of this section.Pf -(E) A FOOD ESTABLISHMENT that PACKAGES cheese using a REDUCED OXYGEN PACKAGING method shall: -(1) Limit the cheeses PACKAGED to those that are commercially manufactured in a FOOD PROCESSING PLANT with no ingredients added in the FOOD ESTABLISHMENT and that meet the Standards of Identity as specified in 21 CFR 133.150 Hard cheeses, 21 CFR 133.169 Pasteurized process cheese or 21 CFR 133.187 Semisoft cheeses; P -(2) Have a HACCP PLAN that contains the information specified under ¶ 8-201.14(D) and as specified under ¶¶ (B)(1), (B)(3)(a), (B)(5) and (B)(6) of this section; Pf -(3) Labels the PACKAGE on the principal display panel with a “use by” date that does not exceed 30 days from its packaging or the original manufacturer’s “sell by” or “use by” date, whichever occurs first; Pf and -(4) Discards the REDUCED OXYGEN PACKAGED cheese if it is not sold for off-PREMISES consumption or consumed within 30 calendar days of its PACKAGING. Pf" Priority Foundation; CDI-Education 0.0
37.3-304.13; Core; Do not use linens or napkins to contact food. 1.0
53.6-501.12; Core; Clean ceiling tile, floors and walls. 0.5
53.6-201.16; Core; Replace ceiling tile that is not approved with smooth, easily cleanable ceiling tile. 0.0
General Comments
- Red Denotes Critical Violation -
- View Last 3 Inspections for this Establishment -
-
- - - - - - - - - - - - - - -
18 Seaboard
- - - - - -
- Location: 18 Seaboard AVE Ste. 100 RALEIGH, NC 27604
- Facility Type: Restaurant
- Inspection Date: 08/27/2014
-
-   -
-
- Score: 97 -
- - - - - - - - - - - - - - - - - - - - - - -
# CommentsPoints
18.3-501.14; Priority; She crab soup and stock were 60 degrees F. and 56 degrees F. Stock had been prepared and cooled two days prior to inspection. She crab soup had a prior day's date for prep. and cooling. Precooked items must be cooled to 45 degrees F or less within six hours. 1.5
20.3-501.16 (A)(2) and (B); Priority; Items like whipped cream cheese were 48.5 degrees F. in reach in cooler. Reach in cooler had an interior temp. of 50 degrees F. Cold foods must be kept at a temp. of 45 degrees F. Do not store potentially hazardous foods in reach in cooler until it is repaired and is maintaining proper cold food storage temperatures. 1.5
General Comments
Keep bottle top covers on drinks by bar hand wash and dish wash sinks. Dust build up was present on shelves by dishwasher. Store raw meats on bottom shelf of freezer.
- Red Denotes Critical Violation -
- View Last 3 Inspections for this Establishment -
-
- - - - - - - - - - - - - - -
1853 Grille
- - - - - -
- Location: 1025 BLUE RIDGE RD RALEIGH, NC 27607-3902
- Facility Type: Restaurant
- Inspection Date: 10/16/2014
-
-   -
-
- Score: 93.5 -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# CommentsPoints
8.5-202.12; Priority Foundation - Hot water at kitchen handwash sink did not exceed 85F after running water for several minutes. Hot water of at least 100 degrees F shall be provided at all handwash sinks. CDI by informing PIC of requirement (NOTE: Hot water at bathroom handwash sink was well above 100F). - 0.0
13.3-304.15 (A); Priority - Observed worker contaminate gloved hands with raw hamburger and then proceeded to handle cooking spatula with contaminated gloves. Single use gloves shall be discarded when soiled with contaminated meat before handling other items in kitchen. CDI by discarding gloves and washing hands (and washing/sanitizing spatula touched). 1.5
14.4-501.114; Priority - Chlorine concentration in one sanitizer spray bottle was a little less than 50 ppm. The chlorine concentration shall be at least 50 ppm but should not exceed 200 ppm. CDI by filling bottle with correct sanitizer strength. - 0.0
20.3-501.16 (A)(2) and (B); Priority - Observed cut lettuce at 49-61F on refrigerated make line top. Cut leafy greens are potentially hazardous and shall be kept at 45/41F or less. CDI by informing PIC of requirement. 0.0
21.3-501.17; Priority Foundation - A few ready-to-eat potentially hazardous foods in refrigeration were not date marked. They shall be date marked if kept more than 24 hours. CDI by informing PIC of requirement. 0.0
23.3-603.11; Priority Foundation - Establishment offers at least one raw or undercooked animal-based food (eggs) and has no consumer advisory. Consumer advisory shall be provided for any raw or undercooked animal-based food with the exception of some steaks. Consumer advisory must consist of a DISCLOSURE, where the menu must indicate where specific foods may be served raw or undercooked, such as "Eggs (cooked to order)," and the disclosure must be asterisked to a REMINDER stating the following: "Consuming raw or undercooked eggs may increase your risk of foodborne illness." CDI by educating PIC about requirements. - 0.0
34.4-502.11(B); Priority Foundation - One metal stemmed food thermometer used in kitchen read 25 degrees F when calibrated using ice-water method. It shall read 32 degrees F, plus or minus 2 degrees F, when calibrated using this method. CDI by properly calibrating this thermometer. 0.5
34.4-302.12; Priority Foundation - No thermometer was available to check temperatures of thin foods. Provide a thermometer with a suitable small diameter probe which is designed to measure the temperature of thin masses such as meat patties and fish fillets. CDI by informing PIC of requirement. - 0.0
36.6-202.15; Core - Observed back door to establishment does not have a self-closing device. Outer openings of a food establishment shall be protected against the entry of insects and rodents by providing self-closing doors. - 1.0
36.6-501.111; Core - Found several live house flies in establishment. Rid the premises of these insects using methods approved for food service establishments. 0.0
36.6-501.112; Core - Insect remains observed on kitchen ceiling light shields. Do not allow insect remains to accumulate. When insect remains are found, document for pest control service and immediately remove. 0.0
37.3-305.12; Core - Found significant ice accumulation on food boxes and packaging stored under leaking condenser in walk-in freezer. Remove ice from boxes and packaging. Food shall not be stored under lines on which water has condensed, or are leaking or have a potential for leaking. Until leaks are fixed, place a pan under leaking areas. - 1.0
43.4-903.11(A) and (C); Core - Observed unprotected disposable food containers, cups and plastic forks with food contact sides up. Some containers were soiled. Discard the soiled containers. Store these containers inverted on clean surfaces with food/beverage sides down. Store forks with handles up and presented to the user. 0.5
45.4-501.11; Core - Replace the wire shelving that is getting rusty and corroded using shelving that is certified or classified for sanitation by an American National Standards Institute (ANSI) accredited certification program, such as the National Sanitation Foundation (NSF) (4-205.10)...Replace the torn walk-in freezer door gasket. 1.0
45.4-205.10; Core - Observed several inverted black soda trays being used as "shelving" to elevate items above floor (in kitchen and walk-in freezer). These trays are not approved as shelves and shall be replace with ANSI/NSF certified shelving. - 0.0
45.4-502.11(A) and (C); Core - Discard the few cracked food containers and replace as needed. 0.0
47.4-601.11(B) and (C); Core - Clean the following soiled surfaces and areas: backsides/undersides of equipment door handles--interiors of refrigerator/freezer units--undersides of sink drainboards--sink drain pipes--bottoms of front counter cabinets--interiors of dining area trash cabinets...Remove stickers from exteriors of food containers...Remove aluminum foil from counters in front battering areas since foil is not easy to clean and sanitize. Consider using large flat sheet pans on counter surfaces instead to catch drips and spills. 0.5
49.5-203.14; Priority - No backflow preventer observed on water supply lines of at least one beverage dispensing machine and ice machine. The plumbing system must be protected from backflow of a solid, liquid or gas contaminant at each point of use on food service equipment by installing an approved backflow prevention device (dual check valves) on water supply lines. CDI by informing PIC of requirement. - 0.0
51.6-202.14; Core - Observed toilet room door is not self-closing. Toilet room door shall be self-closing. 0.0
53.6-501.11; Core - Repair the condensation leak in walk-in freezer so that ice does not accumulate on food boxes and packages below...Replace/repair the cracked caulk between handwash sinks and walls. Replace any moldy or mildewed caulk. When replaced, ensure caulk is smooth and easy to clean...Seal the small holes in kitchen walls...Resecure the loose metal panel on wall behind cooking line so that it is flat against the wall. Seal any cracks or crevices around panel...Replace the burned out ceiling light bulbs to ensure there is at least 50 foot-candles of light on food preparation surfaces (6-303.11). 0.5
53.6-501.12; Core - Clean the soiled walls and flooring in kitchen (especially under equipment and in corners)...Clean the soiled kitchen ceiling light shields, panels and vent openings...Replace stained or discolored panels and light shields. 0.0
54.6-305.11; Core - Observed several personal jackets/sweaters stored on or near dry food items or other food contact items in dry storage areas. Store these personal items in designated areas away from food and other dry storage items. 0.0
General Comments
- Red Denotes Critical Violation -
- View Last 3 Inspections for this Establishment -
-
- Result Pages: - (1) - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 - 11 - 12 - 13 - 14 - 15 - 16 - 17 - 18 - 19 - 20 - 21 - 22 - 23 - 24 - 25 - 26 - 27 - 28 - 29 - 30 - 31 - 32 - 33 - 34 - 35 - 36 - 37 - 38 - 39 - 40 - 41 - 42 - 43 - 44 - 45 - 46 - 47 - 48 - 49 - 50 - 51 - 52 - 53 - 54 - 55 - 56 - 57 - 58 - 59 - 60 - 61 - 62 - 63 - 64 - 65 - 66 - 67 - 68 - 69 - 70 - 71 - 72 - 73 - 74 - 75 - 76 - 77 - 78 - 79 - 80 - 81 - 82 - 83 - 84 - 85 - 86 - 87 - 88 - 89 - 90 - 91 - 92 - 93 - 94 - 95 - 96 - 97 - 98 - 99 - 100 - 101 - 102 - 103 - 104 - 105 - 106 - 107 - 108 - 109 - 110 - 111 - 112 - 113 - 114 - 115 - 116 - 117 - 118 - 119 - 120 - 121 - 122 - 123 - 124 - 125 - 126 - 127 - 128 - 129 - 130 - 131 - 132 - 133 - 134 - 135 - 136 - 137 - 138 - 139 - 140 - 141 - 142 - 143 - 144 - 145 - 146 - 147 - 148 - 149 - 150 - 151 - 152 - 153 - 154 - 155 - 156 - 157 - 158 - 159 - 160 - 161 - 162 - 163 - 164 - 165 - 166 - 167 - 168 - 169 - 170 - 171 - 172 - 173 - 174 - 175 - 176 - 177 - 178 - 179 - 180 - 181 - 182 - 183 - 184 - 185 - 186 - 187 - 188 - 189 - 190 - 191 - 192 - 193 - 194 - 195 - 196 - 197 - 198 - 199 - 200 - 201 - 202 - 203 - 204 - 205 - 206 - 207 - 208 - 209 - 210 - 211 - 212 - 213 - 214 - 215 - 216 - 217 - 218 - 219 - 220 - 221 - 222 - 223 - 224 - 225 - 226 - 227 - 228 - 229 - 230 - 231 - 232 - 233 - 234 - 235 - 236 - 237 - 238 - 239 - 240 - 241 - 242 - 243 - 244 - 245 - 246 - 247 - 248 - 249 - 250 - 251 - 252 - 253 - 254 - 255 - 256 - 257 - 258 - 259 - 260 - 261 - 262 - 263 - 264 - 265 - 266 - 267 - 268 - 269 - 270 - 271 - 272 - 273 - 274 - 275 - 276 - 277 - 278 - 279 - 280 - 281 - 282 - 283 - 284 - 285 - 286 - 287 - 288 - 289 - 290 - 291 - 292 - 293 - 294 - 295 - 296 - 297 - 298 - 299 - 300 - 301 - 302 - 303 -

-
-1212 out of 188163 Inspections Matched Your Query:

Restaurant - in any zipcode - having any violations - between the dates of 08/20/2014 and 12/20/2014
-
-
- -
Query Completed in 0.281 Seconds.
-
-
- - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- Keyword (1):
- -
- -
- Keyword (2):
- -
- -
- Keyword (3):
- -
- -
- Score Range:
- - - -
- Establishment Type:
- -
- Zipcode:
- -
- Violations:
- -
- Date Range:
- - - - - - Start -
- - - - - - End -
- - - - -
- - -
-
-
-
-
-
- -
- - rmfile ./util/resources/nc-wake/searchResultPage.html rmdir ./util/resources/nc-wake hunk ./util/resources/geocode.json 1 -{ - "results" : [ - { - "address_components" : [ - { - "long_name" : "3626", - "short_name" : "3626", - "types" : [ "street_number" ] - }, - { - "long_name" : "Rogers Road", - "short_name" : "Rogers Rd", - "types" : [ "route" ] - }, - { - "long_name" : "Wake Forest", - "short_name" : "Wake Forest", - "types" : [ "locality", "political" ] - }, - { - "long_name" : "Wake Forest", - "short_name" : "Wake Forest", - "types" : [ "administrative_area_level_3", "political" ] - }, - { - "long_name" : "Wake County", - "short_name" : "Wake County", - "types" : [ "administrative_area_level_2", "political" ] - }, - { - "long_name" : "North Carolina", - "short_name" : "NC", - "types" : [ "administrative_area_level_1", "political" ] - }, - { - "long_name" : "United States", - "short_name" : "US", - "types" : [ "country", "political" ] - }, - { - "long_name" : "27587", - "short_name" : "27587", - "types" : [ "postal_code" ] - } - ], - "formatted_address" : "3626 Rogers Road, Wake Forest, NC 27587, USA", - "geometry" : { - "location" : { - "lat" : 35.9446086, - "lng" : -78.50531049999999 - }, - "location_type" : "ROOFTOP", - "viewport" : { - "northeast" : { - "lat" : 35.94595758029151, - "lng" : -78.5039615197085 - }, - "southwest" : { - "lat" : 35.94325961970851, - "lng" : -78.5066594802915 - } - } - }, - "types" : [ "street_address" ] - } - ], - "status" : "OK" -} - rmfile ./util/resources/geocode.json hunk ./util/resources/places.json 1 -{ - "html_attributions" : [], - "results" : [ - { - "geometry" : { - "location" : { - "lat" : 35.94475, - "lng" : -78.505471 - } - }, - "icon" : "http://maps.gstatic.com/mapfiles/place_api/icons/bar-71.png", - "id" : "356a53cace3eb1d7d608bed590039b3f896148de", - "name" : "Il Bacio Italian Grill and Pizzeria", - "opening_hours" : { - "open_now" : true, - "weekday_text" : [] - }, - "photos" : [ - { - "height" : 1200, - "html_attributions" : [], - "photo_reference" : "CnRnAAAAb2YczqG5b46R6vAwWqyXw4c3Xhbke_ndmNRczLezphP5wyJD86m_JRd6kJrSaRfgcz822IanCoQstP9cy7NMGvT8yIIOPNNn9Vy8r3LRz77_kwaBOKe-QdJ8lzmSAEQNAHHWxfnip2W9me_QN7rUXxIQcCTvodOd_9YDnKQvGzHsiBoUvrCVbLNe1vtv_gWjVipxkhYor2g", - "width" : 1600 - } - ], - "place_id" : "ChIJA74XK7xRrIkRIZ37gAtOFFQ", - "price_level" : 2, - "rating" : 3.9, - "reference" : "CpQBhQAAAE5vhsqwQf1urHHnf7RdFOQh0O6yrqy57-7FiV0S7PVJw9lN4zevsuqVUmnnSwVertMsIj52Y0aA1aYAxjx8nnNaSvxwAMcXcvxxQty0_XBD7f1UwoT4VOC5FWKydSAtiDghzf4ZeU_SbAg82xSUvBFuN_OGbYFzCTei4ho_w1fxBoHgb8b_OmUIQQW_bcZcIBIQGXvsRQwp6Ero8W5KPWPirxoUnGmY45pcdH3p_BFAnB6ZFaOtwHM", - "scope" : "GOOGLE", - "types" : [ "bar", "restaurant", "food", "establishment" ], - "vicinity" : "3622 Rogers Road, Wake Forest" - }, - { - "geometry" : { - "location" : { - "lat" : 35.94466, - "lng" : -78.50532 - } - }, - "icon" : "http://maps.gstatic.com/mapfiles/place_api/icons/restaurant-71.png", - "id" : "94f73bedf98c8fc38e4071023c34351a10407cbf", - "name" : "Panda King Chinese Restaurant", - "opening_hours" : { - "open_now" : true, - "weekday_text" : [] - }, - "place_id" : "ChIJ4VjOgb5RrIkRxdtF7HYXhRM", - "price_level" : 2, - "rating" : 4, - "reference" : "CoQBfwAAAAurq_3xmiCGKQXlBxhtvF6fJeBN2LqNn3TjrOT-ZlpBJ_HmemUfM6UOhJNm6GFbGd9m46tHxL3PeAlRHum3JUr3YYlaF0I0a4CsXfAZCSRqg6DorkZTGCRjOYCnflAhk9NWLDqdPB7ZILWWka_StoVeOTak3AVZ8Usp0RC5JVFKEhCUGqUxqYm9siMEH4863-XQGhT6e5_jYUI_hoLQu362JfuN2JUAsw", - "scope" : "GOOGLE", - "types" : [ "restaurant", "food", "establishment" ], - "vicinity" : "3626 Rogers Road, Wake Forest" - } - ], - "status" : "OK" -} rmfile ./util/resources/places.json rmdir ./util/resources hunk ./src/KS/Locate/Places/Geocoding.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{-# LANGUAGE KindSignatures, OverloadedStrings, RankNTypes #-} - -module KS.Locate.Places.Geocoding - ( GeoLatLng (..), forwardLookup ) - where - -import Control.Concurrent ( threadDelay ) -import Data.Aeson -import qualified Data.ByteString.Lazy.Char8 as BL -import Data.Text -import Network.HTTP ( urlEncode ) -import Network.HTTP.Conduit ( simpleHttp ) -import Text.Printf ( printf ) - -import KS.Data.Inspection -import KS.Locate.Locate -import KS.Locate.Config -import KS.Log - - -data GeoLatLng = GeoLatLng Double Double - deriving Show - -instance FromJSON GeoLatLng where - parseJSON (Object v) = do - status <- v .: "status" - when (status /= "OK") $ fail status - - firstResult <- (v .: "results") >>= headE v - loc <- (firstResult .: "geometry") >>= (.: "location") - - GeoLatLng <$> (loc .: "lat") <*> (loc .: "lng") - parseJSON o = fail . show $ o - - -headE :: forall (m :: * -> *) a a1. - (Show a1, Monad m) => a1 -> [a] -> m a -headE _ (x : _) = return x -headE v [] = fail . show $ v - - -forwardLookup :: KSDL GeoLatLng -forwardLookup = do - url <- mkGeocodeUrl - liftIO $ noticeM lname $ "Geocoding URL: " ++ url - - asks (geocodingApiDelay . getConfig) >>= (liftIO . threadDelay) - - gcJSON <- liftIO $ simpleHttp url - liftIO $ debugM lname $ "Geocoding result JSON: " - ++ (BL.unpack gcJSON) - - let parseResult = eitherDecode gcJSON - either - (\status -> throwError $ "ERROR Geocoding: " ++ status) - displayAndReturn parseResult - - -displayAndReturn :: GeoLatLng -> KSDL GeoLatLng -displayAndReturn location = do - liftIO $ noticeM lname $ show location - return location - - -mkGeocodeUrl :: KSDL String -mkGeocodeUrl = do - addr' <- asks (addr . getInspection) - key <- asks (keyString . googleApiKey . getConfig) - - return $ printf "https://maps.googleapis.com/maps/api/geocode/json?address=%s&key=%s" (urlEncode $ unpack addr') key rmfile ./src/KS/Locate/Places/Geocoding.hs hunk ./src/KS/Locate/Places/Match.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{-# LANGUAGE OverloadedStrings #-} -{-# OPTIONS_GHC -fno-warn-unused-do-bind #-} - -module KS.Locate.Places.Match - ( Match, match ) - where - -import Data.Attoparsec.Text hiding ( count, match ) -import Data.Char ( isDigit ) -import Data.Maybe ( catMaybes ) -import qualified Data.Text as T -import qualified Data.Text.Lazy as TL -import qualified Data.Text.Format as TF -import Prelude hiding ( takeWhile ) - -import KS.Locate.Locate -import qualified KS.Data.Inspection as I -import qualified KS.Data.Place as P -import KS.Log - - -type Match = (I.Inspection, P.Place) - -type MatchInternal = (Bool, Match) - - -match :: [P.Place] -> KSDL Match -match ps = do - insp <- asks getInspection - let mis = map (combine insp) ps - let count = (sum . map bToI $ mis) :: Int - - when (count == 0) $ do - throwError "ERROR Match: No Places result matches" - - liftIO $ do - noticeM lname "Matches:" - mapM_ (noticeM lname) $ catMaybes $ map fmtMatched mis - - when (count > 1) $ liftIO $ do - warningM lname "WARNING Match: More than one Places result matched" - - return . head . catMaybes . map positiveMatch $ mis - - where - {- Combine the inspection, places result and a boolean - indicating whether or not we think they refer to the - same place. - - The cleaned-up Places address is returned back to us by - isMatch and substituted into the Place data type here. - -} - combine :: I.Inspection -> P.Place -> MatchInternal - combine insp pl = (matched, (insp, pl { P.vicinity = newPvic })) - where - (matched, newPvic) = - isMatch (I.addr insp) (P.vicinity pl) - - bToI :: MatchInternal -> Int - bToI (True, (_, _)) = 1 - bToI (False, (_, _)) = 0 - - fmtMatched :: MatchInternal -> Maybe String - fmtMatched (True , (_, pl)) = Just . T.unpack . TL.toStrict $ - TF.format "{} | {}" ((P.name pl), (P.vicinity pl)) - fmtMatched (False, (_, _ )) = Nothing - - positiveMatch :: MatchInternal -> Maybe Match - positiveMatch (True , m) = Just m - positiveMatch (False, _) = Nothing - - -{- Determine if two addresses are a "match" based on the beginning - digits. Given how close we get with Google Place search, this - gets us the rest of the way to disambiguate the hits. - - In addition to a True/False match status, we return the cleaned-up - address that was computed below with removePrefixZip. This is - so we can show our users the true address. --} -isMatch :: T.Text -> T.Text -> (Bool, T.Text) -isMatch iaddr pvic = (prefix iaddr == prefix newPvic, newPvic) - where - newPvic = removePrefixZip pvic - prefix = T.takeWhile isDigit - - -{- We get these ridiculous addresses from Google Places where they've - clearly mistakenly put the zip code up front. They look like this: - - "27603, 7900 Fayetteville Road, Raleigh" - - This parser returns the address string with that zip code, - comma and space removed. --} -removePrefixZip :: T.Text -> T.Text -removePrefixZip = - either T.pack id . parseOnly (choice [prefixZip, everythingElse]) - - where - everythingElse :: Parser T.Text - everythingElse = takeWhile $ const True - - prefixZip :: Parser T.Text - prefixZip = do - try $ manyTill digit $ string ", " - everythingElse rmfile ./src/KS/Locate/Places/Match.hs hunk ./src/KS/Locate/Places/NameWords.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{-# LANGUAGE OverloadedStrings #-} - -module KS.Locate.Places.NameWords - ( toList - ) - where - -import qualified Data.List as L -import qualified Data.Map as Map -import Data.Text -import Prelude hiding ( filter, map ) - -import KS.Data.Inspection -import KS.Locate.Config -import KS.Locate.Locate - - -toList :: KSDL [Text] -toList = do - specialCases <- asks (namewordsSpecialCases . getConfig) - list <- mkList - - iname <- asks (name . getInspection) - return $ Map.findWithDefault - list -- Or make a list for a normal name - iname -- Find this name.. - specialCases -- ..in these special cases - - -mkList :: KSDL [Text] -mkList = do - stopwords <- asks (namewordsStopwords . getConfig) - (headList - . L.filter (not . isPrefixOf "#") - . L.filter (\w -> not $ L.elem w stopwords) - . L.take 2 - . split (== ' ') - . remove '\'' - . remove '`' - . remove ',' - . remove '.' - . tr '/' ' ' - . tr '-' ' ' - . toLower - ) `fmap` asks (name . getInspection) - - -headList :: [a] -> [a] -headList (h : _) = [h] -headList _ = [] - - --- Return a string with all of a certain character removed -remove :: Char -> Text -> Text -remove c = filter (not . (== c)) - - --- Transpose a character for another in a Text string -tr :: Char -> Char -> Text -> Text -tr oldCh newCh src = map tr' src where - tr' c - | c == oldCh = newCh - | otherwise = c rmfile ./src/KS/Locate/Places/NameWords.hs hunk ./src/KS/Locate/Places/Places.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{-# LANGUAGE DeriveGeneric, OverloadedStrings #-} -{-# OPTIONS_GHC -fno-warn-unused-binds -fno-warn-orphans #-} - -{-| This module is used for parsing return data from the Google - Places API results. - - To achieve that goal, it contains a custom datatype and JSON - instance, RawPlace, that's not used anywhere else. The reason - for this custom parsing is that the Places API returns a lot - of data we have no interest in. The custom instancing allows - us to discard this unused information. --} - -module KS.Locate.Places.Places - ( coordsToPlaces ) - where - -import Data.Aeson -import qualified Data.ByteString.Lazy.Char8 as BL -import Data.Geospatial ( GeoPoint (..) ) -import qualified Data.List as L -import Data.Text -import GHC.Generics ( Generic ) -import Network.HTTP ( urlEncode ) -import Network.HTTP.Conduit ( simpleHttp ) -import Text.Printf ( printf ) - -import KS.Data.Place -import KS.Locate.Config -import KS.Locate.Locate -import KS.Locate.Places.Geocoding ( GeoLatLng (..) ) -import KS.Locate.Places.NameWords ( toList ) -import KS.Log - - -data RawPlace = RawPlace - { name :: Text - , vicinity :: Text - , location :: GeoPoint - , types :: [String] - , place_id :: Text - } - deriving Generic - -instance FromJSON RawPlace where - parseJSON (Object o) = do - l <- (o .: "geometry") >>= (.: "location") - lng <- l .: "lng" - lat <- l .: "lat" - RawPlace - <$> o .: "name" - <*> o .: "vicinity" - <*> (return . GeoPoint $ [lng, lat]) - <*> o .: "types" - <*> o .: "place_id" - parseJSON o = fail . show $ o - - -newtype Places = Places [RawPlace] - -instance FromJSON Places where - parseJSON (Object v) = do - status <- v .: "status" - when (status /= "OK") $ fail status - - rs <- v .: "results" - when (L.null rs) $ fail . show $ v - - return $ Places rs - parseJSON o = fail . show $ o - - -coordsToPlaces :: GeoLatLng -> KSDL [Place] -coordsToPlaces coords = do - url <- mkPlacesUrl coords - liftIO $ noticeM lname $ "Places URL: " ++ url - - plJSON <- liftIO $ simpleHttp url - liftIO $ debugM lname $ "Places result JSON: " - ++ (BL.unpack plJSON) - - let parseResult = eitherDecode plJSON - either - (\status -> throwError $ "ERROR Places API: " ++ status) - displayAndReturn parseResult - - -convert :: RawPlace -> Place -convert (RawPlace n v l t pid) = Place n v l t pid - - -displayAndReturn :: Places -> KSDL [Place] -displayAndReturn (Places rps) = do - let ps = L.map convert rps - liftIO $ do - noticeM lname "Places returned:" - mapM_ (noticeM lname . show) ps - return ps - - -mkPlacesUrl :: GeoLatLng -> KSDL String -mkPlacesUrl (GeoLatLng lat' lng') = do - key <- asks (keyString . googleApiKey . getConfig) - - nameWords <- toList - liftIO $ noticeM lname $ "Places name words list: " - ++ (show nameWords) - - let nameList = urlEncode $ unpack $ intercalate " " $ nameWords - - searchTypes <- - L.intercalate "|" `fmap` asks (placesTypes . getConfig) - - return $ printf "https://maps.googleapis.com/maps/api/place/nearbysearch/json?key=%s&location=%f,%f&rankby=distance&name=%s&types=%s" key lat' lng' nameList searchTypes rmfile ./src/KS/Locate/Places/Places.hs rmdir ./src/KS/Locate/Places hunk ./src/KS/Locate/Config.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -module KS.Locate.Config - ( Config (..) - , keyString - , loadConfig ) - where - -import qualified Data.Map as Map -import qualified Data.Text as T -import System.Directory ( doesFileExist ) -import System.FilePath -import System.Log -import TCE.Data.ReadConf ( readConfig ) - -import KS.Locate.Opts - - -newtype GoogleKey = GoogleKey String - deriving (Read, Show) - - -data Config = Config - { logPriority :: Priority - , googleApiKey :: GoogleKey - , geocodingApiDelay :: Int - , namewordsStopwords :: [T.Text] - , namewordsSpecialCases :: Map.Map T.Text [T.Text] - , placesTypes :: [String] - } - deriving (Read, Show) - - -loadConfig :: Options -> IO Config -loadConfig options = do - let confPath = (optConfDir options) "ksdl.conf" - conf <- (either error id . readConfig) `fmap` readFile confPath - - -- A Google API key in a file by itself will supercede the one - -- in the conf file - maybe (return conf) (\k -> return $ conf { googleApiKey = k }) - =<< loadGoogleKey options - - --- Google Places API key -loadGoogleKey :: Options -> IO (Maybe GoogleKey) -loadGoogleKey options = do - let keyPath = (optConfDir options) "GoogleAPIKey" - exists <- doesFileExist keyPath - if exists - then do - k <- (GoogleKey . -- ..and construct the proper type - unwords . words) `fmap` -- ..strip any trailing whitespace - (readFile keyPath) - return $ Just k - else return Nothing - - -keyString :: GoogleKey -> String -keyString (GoogleKey ks) = ks rmfile ./src/KS/Locate/Config.hs hunk ./src/KS/Locate/Locate.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -module KS.Locate.Locate - ( Env (..), KSDL, runKSDL - - -- Re-exporting - , asks, liftIO, local, throwError, when - ) - where - -import Control.Monad.Reader -import Control.Monad.Except - -import KS.Data.Inspection -import KS.Locate.Config - - -data Env = Env - { getConfig :: Config - , getInspection :: Inspection - } - -type KSDL a = ReaderT Env (ExceptT String IO) a - -runKSDL :: Env -> KSDL a -> IO (Either String a) -runKSDL env ev = runExceptT (runReaderT ev env) rmfile ./src/KS/Locate/Locate.hs hunk ./src/KS/Locate/Opts.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -module KS.Locate.Opts - ( Options (..) - , defaultOptions - , parseOpts, usageText - ) - where - -import Data.Version ( showVersion ) -import Paths_ks_download ( version ) -import System.Console.GetOpt - - -data Options = Options - { optSuccessDir :: Maybe FilePath - , optFailDir :: Maybe FilePath - , optDelete :: Bool - , optConfDir :: FilePath - , optHelp :: Bool - } - -defaultOptions :: Options -defaultOptions = Options - { optSuccessDir = Nothing - , optFailDir = Nothing - , optDelete = False - , optConfDir = "." - , optHelp = False - } - - -options :: [OptDescr (Options -> Options)] -options = - [ Option ['s'] ["success-dir"] - (ReqArg (\s opts -> opts { optSuccessDir = Just s } ) "SUCCDIR") - "Directory for successful lookups. Optional. If missing, JSON is sent to stdout." - , Option ['f'] ["fail-dir"] - (ReqArg (\s opts -> opts { optFailDir = Just s } ) "FAILDIR") - "Directory for failures. Optional. If missing, failure JSON is not copied anywhere." - , Option [] ["delete"] - (NoArg (\opts -> opts { optDelete = True } )) - "Delete source files as they're processed. BE CAREFUL, this will delete even if no above dest dirs are supplied." - , Option ['c'] ["conf-dir"] - (ReqArg (\s opts -> opts { optConfDir = s } ) "CONFDIR") - "Directory to load ksdl.conf and GoogleAPIKey files from. Defaults to ." - , Option ['h'] ["help"] - (NoArg (\opts -> opts { optHelp = True } )) - "This help text" - ] - - -{- Perform the actual parse of a [String] --} -parseOpts :: [String] -> IO (Options, [String]) -parseOpts args = - case getOpt Permute options args of - (o,n,[] ) -> return (foldl (flip id) defaultOptions o, n) - (_,_,errs) -> ioError $ userError (concat errs ++ usageText) - - -usageText :: String -usageText = (usageInfo header options) ++ "\n" ++ footer - where - header = init $ unlines - [ "Usage: ks-locate [OPTIONS] FILE|DIR" - , "Look up inspections with Google Geocoding and Places" - , "" - , "Options:" - ] - footer = init $ unlines - [ "Looks up the file or dir full of files specified" - , "Writes successful lookups to SUCCDIR or stdout if omitted" - , "Writes failed lookup input files to FAILDIR" - , "Expects to find a ./ksdl.conf file, or at the CONFDIR specified." - , "Logging is written to stdout." - , "" - , "Version " ++ (showVersion version) ++ " Dino Morelli " - ] rmfile ./src/KS/Locate/Opts.hs rmdir ./src/KS/Locate hunk ./src/KS/Database/Mongo.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{-# LANGUAGE OverloadedStrings #-} - -module KS.Database.Mongo - ( parseLastError ) - where - -import Data.Bson -import qualified Data.Text as T -import Text.Printf ( printf ) - - -parseLastError :: Document -> String -parseLastError errdoc = either id id $ lastErrStatus >> atE "err" - where - lastErrStatus :: Either String String - lastErrStatus = case at "ok" errdoc of - 1.0 -> Right "getLastError successful" - c -> Left $ printf "getLastError FAILED: ok: %f" (c :: Double) - - atE :: T.Text -> Either String String - atE key = case at key errdoc of - Just msg -> Left $ printf "insertion FAILED: %s" (T.unpack msg) - Nothing -> Right "insertion successful" rmfile ./src/KS/Database/Mongo.hs rmdir ./src/KS/Database hunk ./src/KS/DLInsp/Source/Downloaders.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -module KS.DLInsp.Source.Downloaders - where - -import qualified Data.Map as M - -import qualified KS.DLInsp.Source.NCWake as NCWake -import KS.DLInsp.Types - - -downloaders :: M.Map String Downloader -downloaders = M.fromList - [ ("nc_wake", NCWake.download) - ] rmfile ./src/KS/DLInsp/Source/Downloaders.hs hunk ./src/KS/DLInsp/Source/NCWake.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -module KS.DLInsp.Source.NCWake - where - -import Data.List ( intercalate, isInfixOf, isPrefixOf ) -import Data.Maybe ( fromMaybe ) -import qualified Data.Text as T -import Data.Time.Calendar ( toGregorian ) ---import Debug.Trace ( trace ) -import Network.HTTP -import Text.HTML.TagSoup -import Text.Printf ( printf ) - -import KS.DLInsp.Types -import qualified KS.Data.Inspection as I - - -urlPrefix :: String -urlPrefix = "http://wake-nc.healthinspections.us/" - - -inspectionSrc :: String -inspectionSrc = "nc_wake" - - -download :: Downloader -download options = runDL options $ do - allPageUrls <- getPageUrls - let pageCount = length allPageUrls - pageLimit <- asks optPageLimit - liftIO $ printf "Downloading %d of %d pages\n\n" - (fromMaybe pageCount pageLimit) pageCount - - let pageUrls = maybe allPageUrls (\n -> take n allPageUrls) pageLimit - - let getters = map getFacilities pageUrls -- [IO [Inspection]] - dir <- asks optDestDir - liftIO $ mapM_ (\ml -> ml >>= mapM_ (I.saveInspection dir)) getters - - --- Get all (4) facilities from a page at the supplied URL -getFacilities :: String -> IO [I.Inspection] -getFacilities url = do - printf "Retrieving %s\n" url - - tags <- parseTags `fmap` (openURL . getRequest $ urlPrefix ++ url) - - let itags = isolateInspTags tags - return $ map extractInsp itags - - --- Extract the block of tags containing each separate facility -isolateInspTags :: [Tag String] -> [[Tag String]] -isolateInspTags= partitions isFacAnchor - where isFacAnchor e = - (e ~== "") && - (isPrefixOf "facilities" $ fromAttrib "href" e) && - (not . elem '&' $ fromAttrib "href" e) - - --- Extract the Inspection data from a facility's tags -extractInsp :: [Tag String] -> I.Inspection -extractInsp tags = I.Inspection - inspectionSrc - (T.pack name) - (T.pack . trim $ addr) - (I.parseDate date) - (read . trim $ score) - violCount - critCount - (reinspToBool reinspection) - (urlPrefix ++ detail) - - where - name = innerText . (takeWhile (not . isTagClose)) $ tags - TagText addr = (dropWhile (~/= "Location:") tags) !! 2 - TagText date = (dropWhile (~/= "Inspection Date:") tags) !! 2 - TagText score = (dropWhile (~/= "Score:") tags) !! 2 - detailTag = head . (dropWhile (~/= "")) $ tags - reinspection = fromAttrib "value" detailTag - detail = extractDetailUrl . fromAttrib "onclick" $ detailTag - - trim = unwords . words - - reinspToBool "Inspection" = False - reinspToBool _ = True - - extractDetailUrl = - takeWhile (/= '\'') . tail . dropWhile (/= '\'') - - - -- This code is for extracting the violations data. Tricky! - - -- Table containing the violations - vtable = (dropWhile (~/= "")) $ tags - - -- Each violations table row. Each of these has 3 tdS - allRowsInViolsTable = partitions (~== "") vtable - - -- Discard the first row, it's display labels - allRowsMinusHeader = tail $ allRowsInViolsTable - - -- Discard the rows after the last violation - vrows = takeWhile isViol allRowsMinusHeader - isViol tags' = length (filter (~== "General Comments") tags') == 0 - - -- Extract the violations and critical-ness from the remaining tags - vs = map extractViolation vrows - - -- Count them up - critCount = length . filter fst $ vs - violCount = length vs - - -{- This code is pulling the violation full text which we are - discarding at this time. But if we want it in the future, it's - already being extracted here. --} -extractViolation :: [Tag String] -> (Bool, String) -extractViolation tags = (crit, text) - where - crit = isInfixOf "red" $ fromAttrib "style" $ tags !! 6 - text = fromTagText $ tags !! 7 - - --- Get the URLs of all search result pages -getPageUrls :: DL [String] -getPageUrls = do - post <- mkPost - tags <- liftIO $ parseTags `fmap` openURL post - return $ map (fromAttrib "href" . head) . - sections (~== "") $ tags - - --- Used for debugging to store the search results page locally --- for inspection -{- FIXME -savePage :: IO () -savePage = do - src <- openURL mkPost - writeFile "temp.html" src --} - - --- Retrieve a page -openURL :: HStream b => Request b -> IO b -openURL p = getResponseBody =<< simpleHTTP p - - -{- These things are for the first form post to get all of the page URLs --} - -mkPost :: DL Request_String -mkPost = do - ps <- searchParams - return $ postRequestWithBody - (urlPrefix ++ "reports.cfm") - "application/x-www-form-urlencoded" - $ intercalate "&" ps - - -searchParams :: DL [String] -searchParams = do - (sy, sm, sd) <- toGregorian `fmap` asks optStartDate - (ey, em, ed) <- toGregorian `fmap` asks optEndDate - - return $ - [ "f=search" - , "strSearch1=" - , "relevance1=fName" - , "strSearch2=" - , "relevance2=fName" - , "strSearch3=" - , "relevance3=fName" - , "lscore=" - , "hscore=" - , "ftype=Restaurant" - , "fzipcode=Any" - , "rcritical=Any" - , "sMonth=" ++ (show sm) - , "sDay=" ++ (show sd) - , "sYear=" ++ (show sy) - , "eMonth=" ++ (show em) - , "eDay=" ++ (show ed) - , "eYear=" ++ (show ey) - , "func=Search" - ] rmfile ./src/KS/DLInsp/Source/NCWake.hs rmdir ./src/KS/DLInsp/Source hunk ./src/KS/DLInsp/Opts.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -module KS.DLInsp.Opts - ( Options (..) - , parseOpts, usageText - ) - where - -import Control.Exception -import Data.List ( intercalate ) -import qualified Data.Map as M -import Data.Time ( Day (..), addDays, fromGregorian, getCurrentTime, - localDay ) -import Data.Time.Zones ( loadLocalTZ, utcToLocalTimeTZ ) -import Data.Version ( showVersion ) -import Paths_ks_download ( version ) -import System.Console.GetOpt -import Text.Regex - -import KS.DLInsp.Source.Downloaders -import KS.DLInsp.Types - - -defaultOptions :: IO Options -defaultOptions = do - tz <- loadLocalTZ -- We expect the TZ env to be set in prod - ut <- getCurrentTime -- This is UTC time - let twoDaysAgo = - addDays (-2) . -- ..two days ago - localDay $ -- ..extract the Day - utcToLocalTimeTZ tz ut -- The local zoned time - - return $ Options - { optSource = "" - , optDestDir = "" - , optStartDate = twoDaysAgo - , optEndDate = twoDaysAgo - , optPageLimit = Nothing - , optHelp = False - } - - -options :: [OptDescr (Options -> Options)] -options = - [ Option ['i'] ["insp-source"] - (ReqArg (\s opts -> opts { optSource = s } ) "SOURCE") - "Inspection source. Required. See SOURCE below." - , Option ['d'] ["dest-dir"] - (ReqArg (\s opts -> opts { optDestDir = s } ) "DESTDIR") - "Directory for downloaded inspection JSON files. Required" - , Option ['s'] ["start-date"] - (ReqArg (\s opts -> opts { optStartDate = parseInputDate s } ) - "YYYYMMDD") - "Starting date for inspection searches. Default: two days ago" - , Option ['e'] ["end-date"] - (ReqArg (\s opts -> opts { optEndDate = parseInputDate s } ) - "YYYYMMDD") - "Ending date for inspection searches. Default: two days ago" - , Option ['l'] ["page-limit"] - (ReqArg (\l opts -> opts { optPageLimit = Just $ read l } ) "PAGES") - "Number of pages to download (applies only to nc_wake?) Default: all of them" - , Option ['h'] ["help"] - (NoArg (\opts -> opts { optHelp = True } )) - "This help text" - ] - - -parseInputDate :: String -> Day -parseInputDate str = - case (matchRegex (mkRegex "([0-9]{4})([0-9]{2})([0-9]{2})") str) of - Just [ys, ms, ds] -> fromGregorian (read ys) (read ms) (read ds) - _ -> throw $ userError $ - "Bad date format: " ++ str ++ "\n" ++ usageText - - --- Perform the args parsing -parseOpts :: [String] -> IO (Options, [String]) -parseOpts args = do - handle ioError $ do - defOpts <- defaultOptions - case getOpt Permute options args of - (o,n,[] ) -> return (foldl (flip id) defOpts o, n) - (_,_,errs) -> throwIO $ userError $ concat errs ++ usageText - - -usageText :: String -usageText = (usageInfo header options) ++ "\n" ++ footer - where - header = init $ unlines - [ "Usage: ks-dlinsp OPTIONS" - , "Acquire inspection data from a source" - , "" - , "Options:" - ] - footer = init $ unlines - [ "Note: If run with no dates, you will get all of the inspections from two days ago. The idea is to give the inspection workers time to get their data into the system and is a good default for daily runs." - , "Logging is written to stdout." - , "" - , "SOURCE is one of: " ++ (intercalate ", " $ M.keys downloaders) - , "" - , "For computing values for 'two days ago', this software will fish out the time zone for your system using the TZ environment variable if possible and /etc/localtime if necessary. On a UTC system (like production on AWS), you need to specify a TZ value from /usr/share/zoneinfo/ like this:" - , "" - , " export TZ=\"America/New_York\"" - , "" - , "Version " ++ (showVersion version) ++ " Dino Morelli " - ] rmfile ./src/KS/DLInsp/Opts.hs hunk ./src/KS/DLInsp/Types.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -module KS.DLInsp.Types - ( Options (..) - , Downloader - , DL, runDL - - -- re-exporting - , asks, liftIO - ) - where - -import Control.Monad.Reader -import Data.Time.Calendar ( Day ) -import System.FilePath () - - -data Options = Options - { optSource :: String - , optDestDir :: FilePath - , optStartDate :: Day - , optEndDate :: Day - , optPageLimit :: Maybe Int - , optHelp :: Bool - } - - -type Downloader = Options -> IO () - - -type DL a = (ReaderT Options IO) a - -runDL :: Options -> DL a -> IO a -runDL env ev = runReaderT ev env rmfile ./src/KS/DLInsp/Types.hs rmdir ./src/KS/DLInsp hunk ./LICENSE 1 -Copyright Dino Morelli 2014-2015 +Copyright Dino Morelli 2015 hunk ./README.md 1 -# kitchensnitch-dl +# ks-library hunk ./README.md 6 -Data downloader and parser for the KitchenSnitch project (Haskell) +Library of common modules for KitchenSnitch (Haskell) hunk ./README.md 17 -### ks-locate - Google Places lookup utility - -`ksdl.conf` and `GoogleAPIKey` files will be looked for by default -in `.` relative to where you ran `ks-locate` from. But it can be -nicer to run from within a directory tree where inspection files -and output files will be written. - -To set up for running like this: - -Make a directory for config files, and copy `ksdl.conf` into it, -along with your Google API key in a file. This can be anywhere as -long as they're together. How about in `$HOME`, like this: - - $HOME/ - .ksdl/ - ksdl.conf - GoogleAPIKey - -The key should be the only thing in the `GoogleAPIKey` file, on a -line by itself. - -Now, to run, have a directory structure like this: - - inspectionsToProcess/ - fail/ - insp/ <-- this is where you have downloaded insp_ json files - succ/ - -Then, to run: - - $ cd inspectionsToProcess - $ ks-locate -c $HOME/.ksdl/ -s succ/ -f fail/ --delete insp | tee ksdl.log - -When it's finished: - - inspectionsToProcess/ - fail/ <-- contains failed insp_ files - insp/ <-- this directory is now empty - succ/ <-- contains successful lookup ks_ files - -Running in this way will allow `ks-locate` to find its conf files -and use relative paths for all those directories, which is nice. - - -### ks-dl-nightly.sh - Utility to run every day to do everything - -For `nc_wake`, no inspections are added on weekend days. The `cron` -job shoulld look something like this: - - 7 1 * * wed,thu,fri,sat,sun /opt/ks-download/bin/ks-dl-nightly.sh - hunk ./README.md 20 -Make sure you have `ghc 7.8.x`, `cabal-install` and `darcs` installed. +Make sure you have `ghc 7.10.x`, `cabal-install` and `darcs` installed. hunk ./README.md 37 -Get the `ks-download` source code - - $ darcs get http://hub.darcs.net/dino/ks-download +Get the `ks-libary` source code + + $ darcs get http://hub.darcs.net/dino/ks-libary hunk ./README.md 66 - -### Building for deployment - -This will build everything into a deployable directory structure -that you can put somewhere like `/opt/` for instance. - - $ cabal configure --prefix=/tmp/ks-download-VER - $ cabal build - $ cabal copy - $ pushd /tmp - $ tar czvf ks-download-VER.tgz ks-download-VER - $ popd +Later, to install into the sandbox (so other project can use this library): + + $ cabal install hunk ./TODO.md 1 -- Do we need ks-docdump and ks-dump any longer? Probably one of them can go. hunk ./TODO.md 3 -- Change mongo password (actually everything), it's insecure now -- [done] Figure out where the missing data is, import everything again to be clear on it -- Organize the new Mongo stuff hunk ./TODO.md 5 -- Split the mongo task up: - - First, convert all of the documents from couch format to mongo, on disk - - Then load all of those and upsert into mongo - -- Triangle potential population of Android users -- Add use case and caching info to the new wiki - - use cases: - - what's around a specific location - - search for a specific restaurant - - caching on the client: - - ?? -- Add msmtp installation notes to wiki - -- Fill out more of README.md -- It may be time to put info in changelog.md -- Look into migrating to MongoDB -- Is there a different url encoding API than what's in Network.HTTP? hunk ./TODO.md 9 -- ks-dlinsp - - Failure to access the URL kills the whole process. Should we handle this with retrying somehow? Not sure - - Should have much more/better logging -- Ks.DlInsp.Source.* - - Replace usage of Network.HTTP with Network.HTTP.Conduit - (http-conduit). See KS.Locate.Places.Geocoding hunk ./doc/dev/notes 1 -## New redesign of the codebase - -- ks-library - - New version of KS.Inspection with changes detailed below - - No longer need the IdInspection type at all - - New version of KS.Locate.Places.Place with changes from KS.Database.Mongo -- ks-download - - raw inspection download (ks-dlinsp) - - Figure out how to eliminate UUIDs from this as well - - What happens: - - main figures out which Downloader to use (based on args) - - Downloader scrapes the web pages - - Constructs a list of KS.Inspection data and calls saveInspection for each - - Google Places location (ks-locate) - - inspection insertion into Mongo (ks-mongo-insert) - - one-off tool for converting KS.Inspection-based json files into modern ones - - No UUID inside - - Filenames change (no UUID) - - date :: UTCTime -- ks-server - - insert on-disk documents (Google Places matched above) into MongoDB - - What happens: - - Iterate over a set of ks...json documents - - Read them into ?? data structures - - Fashion BSON from this - - `save` those into a MongoDB collection - - KS.Database.Mongo contains code from ks-mongo-insert for - de/serializating our new Document with BSON. Also saving - and loading. - - This project will have its own conf file separate from .ksdl or whatever it's called - - src/ - KS/ - Data/ - Document.hs <- KS.Locate.Database.Inspection - Inspection.hs <- KS.Inspection - Place.hs <- KS.Locate.Places.Place - -Modules that model data - -+ KS.Inspection Models the raw inspection before Places matching - Just about everything needs this data type -+ KS.Locate.Database.Inspection Models the data for the nosql *SON document (with save/load) - Uses KS.Inspection, KS.Locate.Places.Match, KS.Locate.Places.Place - Anything that needs to read the ks_.json documents in needs this - This needs a name change. Maybe `DBDocument` or just `Document` - BTW, this has the special FromJSON instancing for Place! Disambiguates it from Places below -- KS.Locate.Places.Geocoding Models GeoLatLng and interfaces to Google Geocoding API -+ KS.Locate.Places.Place Models Google Places data -- KS.Locate.Places.Places Interface and custom parsing of Google Places data - Uses KS.Locate.Places.Place - - -## Importing the data into Mongo - -- The example Mongo introduction stuff is suggesting that we may want to store one document for an entire place, and have all of the inspections in that document. - - This has various implications but the biggest one is that the generated _id will not work any longer. - - The existing _id generation was partly chosen to be workable _before_ the Google Places match - - -Refresher course for the developers, how does the CouchDB solution work? - -- ks-dlinsp calls KS.DLInsp.Source.NCWake.download - - This saves a list of KS.Inspection.IdInspection out - - These are Wake inspections plus the UUID we generate (NOT Places ID) - - Filenames contain the generated UUID -- ks-locate -- /opt/couchbase/bin/cbdocloader - - -Trying to find all of the legacy data (what did you do?) - -on tiddly.honuapps.com - ~dino/ks/imported/ - nc-wake_2014 - nc-wake_2014/ 3000 2014-01-24 - 2015-01-23 - nc-wake_2015-02-09/ 157 2015-02-24 - 2015-02-09 - nc-wake_daily/ - nc-wake_2015-02-10/ 8 - nc-wake_2015-02-11/ 15 - nc-wake_2015-02-12/ 17 - newOne/ - /var/local/kitchensnitch/nc-wake_daily/ - [all dirs] 1352 2015-02-13 - present - -Everything from imported/ above in one dir. This represents all -data before we automated daily downloading - - ~dino/ks/nc-wake_2014-01-24_2015-02-12/ 3179 - -New mongodb server is up-to-date through 2015-07-09. Not yet automated for dailies after that date. - -For purposes of populating a new db, everything we were interested in up to 2015-02-12 is in this file: - - data/nc-wake_2014-01-24_2015-02-12.tgz - -This means no longer mucking around with - - data/nc-wake_2014, data/nc-wake_2014_q1..., data/nc-wake_2015-02-20, etc. - - -## 2015-02-13 - -We are up-to-date with imports to 2015-02-12 - - -## 2015-02-11 - -Nightly download workDirParent: - - /var/local/kitchensnitch/insp_nc_wake - -Make sure this is in the backups! - - -## 2015-02-03 from #haskell - - 13:52 < dino-> quchen: hm, couchbase for Haskell. I could use - that, is it on hub.darcs.net or github? - 13:54 < quchen> dino-: We're still working on it, it's not - finished at all yet :-\ - 13:55 < dino-> quchen: Fantastic that it's being worked on at all! - 13:57 < quchen> dino-: Glad to hear that! Unfortunately I - can't open source it right now ("reasons"), but it's going - to be published under an open license (BSD or something) when - we're done. - - -## Getting and processing a year's worth of inspection data - -So many I did them in 4 blocks for quarters of the year: - -nc-wake data - -nc-wake_q1 2014-01-24 2014-04-23 -nc-wake_q2 2014-04-24 2014-07-23 -nc-wake_q3 2014-07-24 2014-10-23 -nc-wake_q4 2014-10-24 2015-01-23 - - -## 2015-01-15 New Couchbase document structure - -- Is inspection.violations.id an integer or string across different - counties? - - Found violation ids that are alphanumeric in NY state. Example: 2C, 4A -- Any other fields in a violation? - - Yes, the points they cost individually against the whole score for NC - - No, there are no points or even scores for NY -- Is reinspection a boolean or more values? - - -## Isolating subsets of inspection files - -From within some directory full of inspection JSON files.. - - $ cd data/wake_2014.. - -The first 100 - - $ find . type f | head -100 | xargs cp -v -t ../foo/ - -The second 100 - - $ find . type f | head -200 | tail -100 | xargs cp -v -t ../foo/ - -The third 100 - - $ find . type f | head -300 | tail -100 | xargs cp -v -t ../foo/ - - -## 2014-12-29 - -correlating inspection data with location data - -* go through all JSON from health inspections (in data/) -* for each "location", hit GEO CODE API -* get back doc with lat/lng in "geometry" "location" -* pass lat/lng to Places API (REST) -* interested in locations that match on the stree address number -* in those, interested in geometry location and place_id - -Address (from our database) - "location": "4121 NEW BERN AVE STE 109 RALEIGH, NC 27610" - -Geocoder API -- they call this a 'forward lookup' -- given a postal address return the geolocation - - https://maps.googleapis.com/maps/api/geocode/json?address=3D4121%20NEW%20BERN%20AVE%20STE%20109%20RALEIGH,%20NC%2027610 - - "geometry" : { - "location" : { - "lat" : 35.8016467, - "lng" : -78.5653275 - }, - - -Places API: -- use the lat & lng returned by Geolocator to find the restaurant - - https://maps.googleapis.com/maps/api/place/nearbysearch/json?key=APIKEYGOESHERE&location=3D35.8016467,-78.5653275&radius=3D50&types=3Drestaurant - - -* match names -* match location (from our data) to `vicinity` (Places data) -* note the `score` for the string compare using your algorithm -* note the place_id and geometry from the Places API results - - -## first notes - -Starting form is here: http://wake.digitalhealthdepartment.com/reports.cfm - -2015-02: Starting form moved here: http://wake-nc.healthinspections.us/ hunk ./ks-library.cabal 1 -name: ks-download -version: 1.7 -cabal-version: >= 1.8 -build-type: Custom +name: ks-library +version: 1.0 +cabal-version: >= 1.10 +build-type: Simple hunk ./ks-library.cabal 7 -copyright: 2014-2015 Dino Morelli +copyright: 2015 Dino Morelli hunk ./ks-library.cabal 11 -homepage: http://hub.darcs.net/dino/ks-download -synopsis: Data downloader and parser for the KitchenSnitch project +homepage: http://hub.darcs.net/dino/ks-library +synopsis: Library of common modules for KitchenSnitch hunk ./ks-library.cabal 14 -category: Unknown +category: Data hunk ./ks-library.cabal 21 - util/resources/facilities-01 - util/resources/searchResultsPage.html - testsuite/InspectionUUID.hs - testsuite/NameWords.hs - testsuite/test-ks-loc.hs hunk ./ks-library.cabal 24 - location: http://hub.darcs.net/dino/ks-download - -executable ks-dlinsp - main-is: ks-dlinsp.hs + location: http://hub.darcs.net/dino/ks-library + +library + exposed-modules: KS.Data.BSON + KS.Data.Document + KS.Data.Inspection + KS.Data.Place hunk ./ks-library.cabal 33 - , base >= 3 && < 5 - , bytestring - , containers - , filepath - , HTTP - , mtl - , regex-compat - , tagsoup - , text - , time - , tz - , utf8-string - ghc-options: -Wall - -executable ks-locate - main-is: ks-locate.hs - hs-source-dirs: src - build-depends: aeson - , aeson-pretty - , attoparsec - , base >= 3 && < 5 - , bytestring - , containers - , directory - , filepath - , geojson - , hslogger - , HTTP - , http-conduit - , mtl - , old-locale - , regex-compat - , tce-conf - , text - , text-format - , time - , utf8-string - ghc-options: -Wall - -test-suite test-Locate - type: exitcode-stdio-1.0 - main-is: test-Locate.hs - hs-source-dirs: src testsuite - build-depends: aeson - , base >= 3 && < 5 - , bytestring - , containers - , directory - , filepath - , hslogger - , HUnit - , mtl - , regex-compat - , tce-conf - , text - , time - , utf8-string - ghc-options: -Wall - -executable ks-dbinsert - main-is: ks-dbinsert.hs - hs-source-dirs: src - build-depends: aeson - , aeson-pretty - , attoparsec - , base >= 3 && < 5 + , base >= 4.8 && < 4.9 hunk ./ks-library.cabal 36 - , containers hunk ./ks-library.cabal 39 - , hslogger - , HTTP - , http-conduit - , mongoDB - , mtl - , old-locale - , regex-compat - , tce-conf hunk ./ks-library.cabal 40 - , text-format hunk ./ks-library.cabal 41 - , utf8-string hunk ./ks-library.cabal 42 - -test-suite test-Database - type: exitcode-stdio-1.0 - main-is: test-Database.hs - hs-source-dirs: src testsuite - build-depends: base >= 3 && < 5 - , bson - , HUnit - , text - ghc-options: -Wall - ---executable ks-dump --- main-is: ks-dump.hs --- hs-source-dirs: src --- build-depends: aeson --- , aeson-pretty --- , attoparsec --- , base >= 3 && < 5 --- , bytestring --- , containers --- , directory --- , filepath --- , hslogger --- , HTTP --- , http-conduit --- , mtl --- , old-locale --- , regex-compat + default-language: Haskell2010 + +--test-suite test-Database +-- type: exitcode-stdio-1.0 +-- main-is: test-Database.hs +-- hs-source-dirs: src testsuite +-- build-depends: base >= 3 && < 5 +-- , bson +-- , HUnit hunk ./ks-library.cabal 52 --- , text-format --- , time --- , utf8-string --- ghc-options: -Wall - ---executable ks-docdump --- main-is: ks-docdump.hs --- hs-source-dirs: src --- build-depends: aeson --- , aeson-pretty --- , attoparsec --- , base >= 3 && < 5 --- , bytestring --- , containers --- , directory --- , filepath --- , hslogger --- , HTTP --- , http-conduit --- , mtl --- , old-locale --- , regex-compat --- , tce-conf --- , text --- , text-format --- , time --- , utf8-string hunk ./ksdl.conf 1 -Config - -- Logging Priority - -- DEBUG - very fine debug info - -- INFO - less fine debug info - -- NOTICE - normal runtime info (lots!), a reasonable default - -- WARNING - warnings that don't affect lookups - -- ERROR - non-fatal runtime errors, missing lookups for ex - -- CRITICAL - -- ALERT - -- EMERGENCY - fatal runtime errors - - --{ logPriority = DEBUG - --{ logPriority = INFO - { logPriority = NOTICE - - , googleApiKey = GoogleKey "foobarbaz" - - -- Geocoding API limit: 2500/day, 5/sec - -- This field in microseconds - , geocodingApiDelay = 250000 -- 0.25s - - -- When we make the Places API call name= list, discard these words - , namewordsStopwords = - [ "" - , "&" - , "@" - , "a" - , "and" - , "at" - , "bagel" -- These look like a bad idea but all of the many - , "bagels" -- Brueggers Bagels sites are broken without them. - , "bakery" - , "bar" - , "barbecue" - , "by" - , "cafe" - , "cafeteria" - -- , "cuisine" -- Not sure about this one yet - , "diner" - , "grill" - , "grille" - , "in" - , "italian" - , "japanese" - , "nc" - , "new" - , "north" - , "of" - , "on" - , "pizza" - , "rest" - , "restaurant" - , "rtp" - , "sub" - , "subs" - , "the" - ] - -- Special handling of name words list for some businesses - , namewordsSpecialCases = fromList - -- These are for nc_wake: NC, Wake county - [ ("1250 Heritage", ["heritage"]) - , ("1705 Prime Catering and Events", ["catering"]) - , ("Ãn", ["an"]) - , ("Andres` Coffee & Crepes", ["coffee"]) - , ("Blue Ridge - The Museum Restaurant", ["iris"]) - , ("Bojangles#37", ["bojangles"]) - , ("BOTTLE MIXX", ["bottlemixx"]) - , ("BUKU", ["bu•ku"]) - , ("BURGER FI", ["burgerfi"]) - , ("BURGER FI #0002", ["burgerfi"]) - , ("BREWMASTER BAR & GRILL", ["brewmasters"]) - , ("Brueggers`s Bagel ", ["brueggers"]) - , ("Caffe Mars/Catering on Demand", ["catering"]) - , ("CALAVERA EMPANADA & TEQUILA BAR", ["cala"]) - , ("D & S CAFETERIA", ["d&s"]) - , ("Delightful Inspirations", ["delightfull"]) - , ("DeM0'S PIZZERIA & DELI", ["demos"]) - , ("DREAMSPORTS CENTER", ["dream"]) - , ("Earthfare The Healthy Supermarket Restaurant", ["earth"]) - , ("Fuji San Japanese Steakhouse", ["fujisan"]) - , ("G G FISH AND CHICKEN", ["gg"]) - , ("Homegrown Pizza", ["home"]) - , ("Hong Kong Restaurant", ["restaurant"]) - , ("HONG KONG CHINESE KITCHEN", ["chinese"]) - , ("INTER-FAITH FOOD SHUTTLE @ INTERACT", ["interact"]) - , ("International House Of Pancakes", ["ihop"]) - , ("International House of Pancakes #408", ["ihop"]) - , ("ITALIAN KITCHEN", ["italian"]) - , ("Jelly Beans Rest", ["jellybeans"]) - , ("J J CHINA", ["jj"]) - , ("Joy Luck Of Grand Asia Market", ["asia"]) - , ("K & W Cafeteria", ["k&w"]) - , ("KICK BACK JACK`S", ["kickback"]) - , ("LaForesta Italian Cafe And Pizzeria", ["la"]) - , ("Lonestar Steakhouse", ["lone"]) - , ("MCLEAN`S HILLBILLIES", ["hillbillies"]) - , ("McLean`s Ole Time Cafe", ["ole"]) - , ("Milano`s Pizza", ["milano"]) - , ("No Fo At The Pig", ["nofo"]) - , ("NO. 1 CHINESE RESTAURANT", ["chinese"]) - , ("N.Y.Pizza", ["pizza"]) - , ("NYBD III", []) - , ("THE ORIGINAL NY PIZZA", ["ny"]) - , ("PARK WEST 14 CINEMAS", ["14"]) - , ("Pho2 Far East", ["pho"]) - , ("Piccola Italia", ["piccola"]) - , ("Pizza Inn", ["pizza"]) - , ("POLAR ICE HOUSE", ["cary"]) - , ("THE PR AT THE POOLE", ["poole"]) - , ("RALLYPOINT SPORT GRILL", ["rally"]) - , ("Ralph`s Grocery & Deli/Grill", ["ralph"]) - , ("Romas Pizzeria", ["roma"]) - , ("Shaba Shabu", ["shabashabu"]) - , ("S-MART CAFE", ["smart"]) - , ("SALT& LIME CABO GRILL", ["salt"]) - , ("SHISH KABOB AT PINECREST", ["shishkabob"]) - , ("Super Target Food Avenue", ["target"]) - , ("Super Target Store T1826 (Food Ave)", ["target"]) - , ("Taqueria Rancho Grande", ["rancho"]) - , ("TERRA FIN STATION", ["terrafin"]) - , ("Z Pizza", ["zpizza"]) - , ("Z PIZZA CARY", ["zpizza"]) - ] - - , placesTypes = - [ "restaurant" - , "food" - , "establishment" - --, "cafe" - --, "bar" - ] - } rmfile ./ksdl.conf hunk ./src/KS/Log.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -module KS.Log - ( initLogging, lname - , logStartMsg, logStopMsg - , line - - -- Re-exported from System.Log - , Priority (..), debugM, infoM, noticeM, warningM, errorM - , criticalM, alertM, emergencyM - ) - where - -import Data.Time ( defaultTimeLocale, formatTime, getCurrentTime - , utcToLocalZonedTime ) -import System.IO ( stdout ) -import System.Log.Handler.Simple ( streamHandler ) -import System.Log.Logger - - -lname :: String -lname = "normal-output" - - -{- Set up logging --} -initLogging :: Priority -> IO () -initLogging logPriority = do - -- Remove the root logger's default handler that writes every - -- message to stderr! - updateGlobalLogger rootLoggerName removeHandler - - -- Set up our logger - h <- streamHandler stdout DEBUG - updateGlobalLogger lname $ setHandlers [h] - updateGlobalLogger lname $ setLevel logPriority - - -logStartMsg :: String -> IO () -logStartMsg name = do - l <- getLogger name - let displayLevel = maybe "NO LEVEL SET" show $ getLevel l - timeString <- timeStamp - noticeM name $ timeString ++ " Logging started, Priority: " - ++ displayLevel - - -logStopMsg :: String -> IO () -logStopMsg name = do - timeString <- timeStamp - noticeM name $ timeString ++ " Logging stopped" - - -timeStamp :: IO String -timeStamp = do - local <- utcToLocalZonedTime =<< getCurrentTime - return $ formatTime defaultTimeLocale "%F %T %Z" local - - -line :: String -line = replicate 70 '-' rmfile ./src/KS/Log.hs hunk ./src/ks-dbinsert.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{-# LANGUAGE OverloadedStrings #-} - -{- This is for inserting inspections into MongoDB --} - -import Control.Monad ( (>=>) ) -import Data.List ( isPrefixOf ) -import Database.MongoDB -import System.Directory ( doesFileExist, getDirectoryContents ) -import qualified Data.Text as T -import System.Environment ( getArgs ) -import System.FilePath -import System.IO - ( BufferMode ( NoBuffering ) - , hSetBuffering, stdout, stderr - ) -import Text.Printf ( printf ) - -import KS.Data.BSON ( docToBSON ) -import qualified KS.Data.Document as D -import KS.Database.Mongo ( parseLastError ) - - -{- Some of this goes into config --} -mongoServerIP :: String -mongoServerIP = "tiddly.honuapps.com" - -m_host :: Host -m_host = host mongoServerIP ---m_host = Host mongoServerIP notDefaultPort - -m_db, m_collection, m_user, m_pass :: T.Text -m_db = "ks" ---m_collection = "insp_indiv" -- with UUID ---m_collection = "insp_objid" -- without UUID -m_collection = "insp_test" -- development -m_user = "mongoks" -m_pass = "vaiDae8z" - - -main :: IO () -main = do - -- No buffering, it messes with the order of output - mapM_ (flip hSetBuffering NoBuffering) [ stdout, stderr ] - - (command : srcDirOrFile : rest') <- getArgs - - -- Paths to all files we'll be processing - files <- buildFileList srcDirOrFile - - case command of - --"getone" -> withDB getOne - --"convertname" -> -- Just a name change using saveDoc - {- - "convert" -> do - let (outDir : _) = rest' - mapM_ (saveNewFormat outDir) files - -} - "display" -> mapM_ (D.loadDoc >=> print) files - "insert" -> withDB (\p -> mapM_ (loadAndInsert p) files) - _ -> undefined - - -withDB :: (Pipe -> IO ()) -> IO () -withDB action = do - -- Get a connection to Mongo, they call it a 'pipe' - pipe <- connect m_host - - -- Authenticate with mongo, show the auth state on stdout - (access pipe UnconfirmedWrites m_db $ auth m_user m_pass) >>= - \tf -> putStrLn $ "Authenticated with Mongo: " ++ (show tf) - - action pipe - - close pipe - - -{- FIXME Broken by abandonment of UUID -getOne :: Pipe -> IO () -getOne pipe = do - return () - d <- access pipe UnconfirmedWrites m_db $ do - let muuid = UUID . toASCIIBytes . fromJust . fromString - $ "ec53d9d7-c8e8-553b-9328-c10d6908a43b" - rest =<< find (select ["_id" =: muuid] m_collection) - print d --} - - -getAll :: Pipe -> IO () -getAll pipe = do - is <- access pipe UnconfirmedWrites m_db $ do - rest =<< find (select [] m_collection) - mapM_ print is - - -loadAndInsert :: Pipe -> FilePath -> IO () -loadAndInsert pipe path = do - edoc <- D.loadDoc path - - result <- case edoc of - Left errMsg -> return errMsg - Right doc -> access pipe UnconfirmedWrites m_db $ do - save m_collection $ docToBSON doc - parseLastError `fmap` runCommand [ "getLastError" =: (1::Int) ] - - printf "%s %s\n" path result - - -buildFileList :: FilePath -> IO [FilePath] -buildFileList srcDirOrFile = do - isFile <- doesFileExist srcDirOrFile - if isFile then return [srcDirOrFile] - else - ( map (srcDirOrFile ) -- ..relative paths - . filter (not . isPrefixOf ".") ) -- ..minus dotfiles - `fmap` getDirectoryContents srcDirOrFile -- All files rmfile ./src/ks-dbinsert.hs hunk ./src/ks-dlinsp.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -import Control.Monad ( when ) -import qualified Data.Map as M -import System.Environment ( getArgs ) -import System.Exit ( exitFailure, exitSuccess ) -import System.IO - ( BufferMode ( NoBuffering ) - , hSetBuffering, stdout, stderr - ) - -import KS.DLInsp.Opts -import KS.DLInsp.Source.Downloaders - - -main :: IO () -main = do - -- No buffering, it messes with the order of output - mapM_ (flip hSetBuffering NoBuffering) [ stdout, stderr ] - - (options, _) <- getArgs >>= parseOpts - - when (optHelp options) $ do - putStrLn usageText - exitSuccess - - when (optDestDir options == "") $ do - putStrLn usageText - exitFailure - - let mbDownloader = M.lookup (optSource options) downloaders - maybe (putStrLn usageText >> exitFailure) - (\dl -> dl options) mbDownloader rmfile ./src/ks-dlinsp.hs hunk ./src/ks-docdump.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{- This is a development tool for exploring the data in various ways --} - -import Control.Monad ( (>=>) ) -import Data.List ( isPrefixOf ) -import System.Directory ( doesFileExist - , getDirectoryContents ) -import qualified Data.Text as T -import System.Environment ( getArgs ) -import System.FilePath -import System.IO - ( BufferMode ( NoBuffering ) - , hSetBuffering, stdout, stderr - ) - -import qualified KS.Inspection as I -import KS.Locate.Database.Inspection ( Document (..), loadDoc ) - - -main :: IO () -main = do - -- No buffering, it messes with the order of output - mapM_ (flip hSetBuffering NoBuffering) [ stdout, stderr ] - - (srcDirOrFile : _) <- getArgs - - -- Paths to all files we'll be processing - files <- buildFileList srcDirOrFile - - -- Load and display each document - mapM_ (loadDoc >=> display) files - - -buildFileList :: FilePath -> IO [FilePath] -buildFileList srcDirOrFile = do - isFile <- doesFileExist srcDirOrFile - if isFile then return [srcDirOrFile] - else - ( map (srcDirOrFile ) -- ..relative paths - . filter (not . isPrefixOf ".") ) -- ..minus dotfiles - `fmap` getDirectoryContents srcDirOrFile -- All files - - -display :: Either String Document -> IO () -display (Left msg) = print msg -display (Right doc) = putStrLn - $ (_id doc) ++ " | " - ++ (T.unpack . I.name . inspection $ doc) rmfile ./src/ks-docdump.hs hunk ./src/ks-dump.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{- This is a development tool for exploring the data in various ways --} - -import Control.Monad ( (>=>) ) -import Data.List ( isPrefixOf ) -import System.Directory ( doesFileExist - , getDirectoryContents ) -import qualified Data.Text as T -import System.Environment ( getArgs ) -import System.FilePath -import System.IO - ( BufferMode ( NoBuffering ) - , hSetBuffering, stdout, stderr - ) - -import KS.Inspection - - -main :: IO () -main = do - -- No buffering, it messes with the order of output - mapM_ (flip hSetBuffering NoBuffering) [ stdout, stderr ] - - (srcDirOrFile : _) <- getArgs - - -- Paths to all files we'll be processing - files <- buildFileList srcDirOrFile - - -- Look up each inspection with Geocoding and Places - mapM_ (loadInspection' >=> display) files - - -buildFileList :: FilePath -> IO [FilePath] -buildFileList srcDirOrFile = do - isFile <- doesFileExist srcDirOrFile - if isFile then return [srcDirOrFile] - else - ( map (srcDirOrFile ) -- ..relative paths - . filter (not . isPrefixOf ".") ) -- ..minus dotfiles - `fmap` getDirectoryContents srcDirOrFile -- All files - - -loadInspection' :: FilePath -> IO IdInspection -loadInspection' path = do - parseResult <- loadInspection path - either - (\msg -> error $ "ERROR Inspection: " ++ path ++ "\n" ++ msg) - return parseResult - - -display :: IdInspection -> IO () -display (IdInspection _id' insp) = putStrLn $ - (show . date $ insp) ++ " | " ++ - (T.unpack . name $ insp) ++ " | " ++ - _id' -{- -display :: IdInspection -> IO () -display (IdInspection _id' insp) = putStrLn $ - (T.unpack . name $ insp) ++ " | " ++ - (T.unpack . addr $ insp) ++ " | " ++ - _id' --} rmfile ./src/ks-dump.hs hunk ./src/ks-locate.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -import Data.Aeson.Encode.Pretty hiding ( Config ) -import qualified Data.ByteString.Lazy.Char8 as BL -import Data.List ( isPrefixOf ) -import System.Directory ( copyFile, doesFileExist - , getDirectoryContents, removeFile ) -import System.Environment ( getArgs ) -import System.Exit ( exitSuccess ) -import System.FilePath -import System.IO - ( BufferMode ( NoBuffering ) - , hSetBuffering, stdout, stderr - ) - -import KS.Data.Document ( Document (..), saveDoc ) -import KS.Data.Inspection -import KS.Locate.Config -import KS.Locate.Locate -import KS.Locate.Opts -import KS.Locate.Places.Geocoding ( forwardLookup ) -import KS.Locate.Places.Match ( Match, match ) -import KS.Locate.Places.Places ( coordsToPlaces ) -import KS.Log - - -main :: IO () -main = do - -- No buffering, it messes with the order of output - mapM_ (flip hSetBuffering NoBuffering) [ stdout, stderr ] - - (options, srcDirsOrFiles) <- getArgs >>= parseOpts - when ((optHelp options) || (null srcDirsOrFiles)) $ do - putStrLn usageText - exitSuccess - - -- Load the config file - config <- loadConfig options - - initLogging $ logPriority config - logStartMsg lname - - -- Paths to all files we'll be processing - files <- concat `fmap` - (sequence $ map buildFileList srcDirsOrFiles) - - -- Look up each inspection with Geocoding and Places - mapM_ (lookupInspection config options) files - - noticeM lname line - - logStopMsg lname - - -buildFileList :: FilePath -> IO [FilePath] -buildFileList srcDirOrFile = do - isFile <- doesFileExist srcDirOrFile - if isFile then return [srcDirOrFile] - else - ( map (srcDirOrFile ) -- ..relative paths - . filter (not . isPrefixOf ".") ) -- ..minus dotfiles - `fmap` getDirectoryContents srcDirOrFile -- All files - - -lookupInspection :: Config -> Options -> FilePath -> IO () -lookupInspection config options srcPath = do - r <- runKSDL (Env config nullInspection) $ do - liftIO $ noticeM lname line - - insp <- loadInspection' srcPath - local (\r -> r { getInspection = insp }) $ do - geo <- forwardLookup - places <- coordsToPlaces geo - match places - - either (handleFailure) (outputDoc options srcPath . mkDoc) r - - where - handleFailure msg = do - -- Copy to FAILDIR if we have one - maybe (return ()) (\failDir -> - copyFile srcPath $ failDir takeFileName srcPath) - $ optFailDir options - - -- Delete the original if we've been instructed to do so - when (optDelete options) $ removeFile srcPath - - -- Log what happened - errorM lname msg - - mkDoc :: Match -> Document - mkDoc (inspection', place') = - Document "inspection" inspection' place' - - -outputDoc :: Options -> FilePath -> Document -> IO () -outputDoc options srcPath doc = do - r <- case (optSuccessDir options) of - Just successDir -> saveDoc successDir doc - Nothing -> do - BL.putStrLn $ encodePretty doc - return $ Right () - - case r of - Left msg -> putStrLn msg - Right () -> when (optDelete options) $ removeFile srcPath - - -loadInspection' :: FilePath -> KSDL Inspection -loadInspection' path = do - parseResult <- liftIO $ loadInspection path - either - (\msg -> throwError $ "ERROR Inspection: " ++ path ++ "\n" ++ msg) - (\insp -> (liftIO $ noticeM lname $ show insp) >> return insp) - parseResult rmfile ./src/ks-locate.hs hunk ./testsuite/Mongo.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{-# LANGUAGE OverloadedStrings #-} - -module Mongo - ( tests ) - where - -import Data.Bson -import Data.Text hiding ( map ) -import Test.HUnit - -import KS.Database.Mongo - - -tests :: Test -tests = TestList $ map testLastError testData - - -testLastError :: (String, Document, String) -> Test -testLastError (label', doc, expected) = TestCase $ do - let actual = parseLastError doc - assertEqual label' expected actual - - -testData :: [(String, Document, String)] -testData = - [ ( "getLastError ok, db operation ok" - , [ "connectionId" =: (652 :: Int) - , "n" =: (0 :: Int) - , "syncMillis" =: (0 :: Int) - , "writtenTo" =: (Nothing :: Maybe Text) - , "err" =: (Nothing :: Maybe Text) - , "ok" =: (1.0 :: Double) - ] - , "insertion successful" - ) - , ( "getLastError ok, db operation NOT ok" - , [ "connectionId" =: (652 :: Int) - , "n" =: (0 :: Int) - , "syncMillis" =: (0 :: Int) - , "writtenTo" =: (Nothing :: Maybe Text) - , "err" =: ((Just "Some error occurred with insertion!") :: Maybe Text) - , "ok" =: (1.0 :: Double) - ] - , "insertion FAILED: Some error occurred with insertion!" - ) - , ( "getLastError NOT ok" - , [ "connectionId" =: (652 :: Int) - , "n" =: (0 :: Int) - , "syncMillis" =: (0 :: Int) - , "writtenTo" =: (Nothing :: Maybe Text) - , "err" =: ((Just "Should never see this") :: Maybe Text) - , "ok" =: (2.0 :: Double) - ] - , "getLastError FAILED: ok: 2.0" - ) - ] rmfile ./testsuite/Mongo.hs hunk ./testsuite/NameWords.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -{-# LANGUAGE OverloadedStrings #-} - -module NameWords - ( tests ) - where - -import Data.Text hiding ( map ) -import Test.HUnit -import Text.Printf ( printf ) - -import KS.Data.Inspection -import KS.Locate.Config -import KS.Locate.Locate -import KS.Locate.Opts -import KS.Locate.Places.NameWords ( toList ) - - -tests :: Test -tests = TestList $ map testNameWords testData - - -testNameWords :: (Text, [Text]) -> Test -testNameWords (input, output) = TestCase $ do - conf <- loadConfig defaultOptions - let insp = fakeInspection input - actual <- runKSDL (Env conf insp) toList - let label = printf "name words for \"%s\"" (unpack input) - assertEqual label (Right output) actual - - -testData :: [(Text, [Text])] -testData = - [ ("Belle at The Jones House", ["belle"]) - , ("BOJANGLES #15", ["bojangles"]) - , ("Cafe Tiramisu/North Ridge Pub", ["tiramisu"]) - , ("Flights Restaurant-Raleigh Renaissance", ["flights"]) - , ("INTER-FAITH FOOD SHUTTLE @ INTERACT", ["interact"]) - , ("K&W CAFETERIA", ["k&w"]) - , ("Kadhai-The Indian Wok", ["kadhai"]) - , ("NC Farm Bureau Cafeteria", ["farm"]) - , ("New Wangs Kitchen", ["wangs"]) - , ("Piccola Italia", ["piccola"]) - , ("Quiznos Sub", ["quiznos"]) - , ("R.J.`S PLACE", ["rjs"]) - , ("SAMI'S SUBS, PIZZA & MORE", ["samis"]) - , ("Tonys Bourbon Street Oyster Bar", ["tonys"]) - ] - - -fakeInspection :: Text -> Inspection -fakeInspection name' = nullInspection { name = name' } rmfile ./testsuite/NameWords.hs hunk ./testsuite/test-Database.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -import System.Exit -import Test.HUnit hiding ( counts ) - -import qualified Mongo - - -main :: IO () -main = do - counts <- runTestTT tests - exit $ testsPassed counts - - -exit :: Bool -> IO () -exit True = exitWith ExitSuccess -exit False = exitWith $ ExitFailure 1 - - -testsPassed :: Counts -> Bool -testsPassed (Counts _ _ e f) = (e == 0) && (f == 0) - - -tests :: Test -tests = TestList - [ Mongo.tests - ] rmfile ./testsuite/test-Database.hs hunk ./testsuite/test-Locate.hs 1 --- License: BSD3 (see LICENSE) --- Author: Dino Morelli - -import System.Exit -import Test.HUnit hiding ( counts ) - -import qualified NameWords - - -main :: IO () -main = do - counts <- runTestTT tests - exit $ testsPassed counts - - -exit :: Bool -> IO () -exit True = exitWith ExitSuccess -exit False = exitWith $ ExitFailure 1 - - -testsPassed :: Counts -> Bool -testsPassed (Counts _ _ e f) = (e == 0) && (f == 0) - - -tests :: Test -tests = TestList - [ NameWords.tests - ] rmfile ./testsuite/test-Locate.hs hunk ./util/inspconv.coffee 1 -fs = require 'fs' -#{ puts } = require 'util' - - -inDir = "insp" -outDir = "insp-fix" - -fs.mkdirSync outDir, 0o0755 unless fs.existsSync outDir - -fileNames = fs.readdirSync inDir - - -transform = (fileName) -> - insp = JSON.parse fs.readFileSync "#{inDir}/#{fileName}" - - _id = insp._id # extract the _id field - delete insp._id # and then remove it - - # Write the new file - fs.writeFileSync "#{outDir}/#{fileName}.json", - # Put _id and remaining data back together new IdInspection shape - JSON.stringify { "_id": "#{_id}", "inspection": insp } - - -transform fileName for fileName in fileNames rmfile ./util/inspconv.coffee hunk ./util/ks-dl-nightly.sh 1 -#! /bin/bash - -# Location of the ks-download binaries and scripts -# Needed for the cron environment -binDir=/opt/ks-download/bin -#binDir=/home/dino/bin - -# Make this a directory that's backed-up -workDirParent=/var/local/kitchensnitch/nc-wake_daily -#workDirParent=/home/dino/dev/kitchensnitch/data/nc-wake_daily - -# Couchbase password -couchPassword=COUCH_PASSWORD - - -# When running outside of this zone (or on a system using UTC), -# need this set explicitly for both the `date` binary and ks-dlinsp -export TZ="America/New_York" - -workDir=${workDirParent}/nc-wake_$(date +"%Y-%m-%d" --date='2 days ago') - -PATH=$binDir:"${PATH}" - - -mkdir -p $workDir/{insp,succ,fail} -cd $workDir - - -# Scrape new inspections from two days ago - -ks-dlinsp \ - --insp-source nc_wake \ - --dest-dir insp \ - > ks-dlinsp.log - - -# Places match the inspections - -ks-locate \ - --conf-dir /home/dino/.ksdl \ - --success-dir succ \ - --fail-dir fail \ - --delete \ - insp \ - > ks-locate.log - - -# Import into Couchbase - -#/opt/couchbase/bin/cbdocloader -u Administrator -p "$couchPassword" -n localhost:8091 -b kitchen_snitch succ/ > ks-couch-import.log 2>&1 - -# Use these to simulate import success or failure -# comment these lines out if using the above cbdocloader command -echo "Fake cbdocloader success" > ks-couch-import.log; true -#echo "Fake cbdocloader failure" > ks-couch-import.log; false - -couchExit=$? - -if [ $couchExit != 0 ] -then - echo "There was a problem importing KS records from ${workDir}/succ/" - echo "cbdocloader exit code: $couchExit" - echo "see ${workDir}/ks-couch-import.log" -fi rmfile ./util/ks-dl-nightly.sh