ddgsearch.py — search duckduckgo, scrape the results, in python
I’ve been writing a series of articles about using OpenAI’s GPT (currently the model text-davinci-003) to summarize youtube videos, and other kinds of documents, and to interactively chat about those documents.
I’ve got plans. Plans that involve my GPT-based chatbots having access to the internet to get more information. But how do I do that?
An essential step is to be able to search the web, get search results, and download useful text from the urls in the search results.
So in this article, I’m presenting a python module, ddgsearch.py, which can do this. You can find it here:
You can call it from the cli like this:
python ddgsearch.py "how to make a great pastrami sandwich" --numresults 2 --clean_with_llmand the result will look like this (note that this is the truncated output to the console; the full output is written to a file):
***************************************
Result 1
Url: https://www.delish.com/cooking/a37720/pastrami-sandwiches/
Title: Pastrami Sandwich Recipe - Best Pastrami Sandwiches
Cleaned Text (shortened):
The text extracted from the webpage contains information about different variations of the classic pastrami sandwich. It mentions recipes for a New York Reuben, a French Dip, and a Pastrami Melt. It also provides tips on how to make the perfect pastrami sandwich, such as using the right bread and adding condiments.
-Substituting pastrami for corned beef in a "New Yorker" version of a Big Apple deli favorite.
-Pastrami Burgers featuring spiced beef and mustard on rye toast.
-Turkey Pastrami Sandwich with Swiss cheese, sauerkraut and apple.
-Hot Steamed Pastrami on perfect rye bread.
-Mini Triple-Decker Pastrami Reuben Sandwiches.
-Top Chef Stefan Richter's Pastrami on Rye.
Pastrami Sandwich Recipe:
- Top Chef Stefan Richter creates the ultimate pastrami on rye, adding Bavaria blue cheese and homemade horseradish sauce to this classic New York sandwich.
- Lisa Linder's open-faced pastrami sandwiches are topped with bright condiments like braised red cabbage and green cabbage in sour-cream sauce.
***************************************
***************************************
Result 2
Url: https://pocketchangegourmet.com/pastrami-sandwich/
Title: How to Make a Classic Pastrami Sandwich
Cleaned Text (shortened):
The Best Pastrami Sandwich Recipe:
Ingredients:
- Rye bread
- Pastrami
- Mustard
- Pickles
Instructions:
1. Preheat oven to 350°F.
2. Place rye bread slices on a baking sheet and bake for 10 minutes.
3. Heat pastrami in a skillet over medium heat for 5 minutes.
4. Spread mustard on one side of each slice of bread.
5. Place pastrami on one slice of bread and top with pickles.
6. Place the other slice of bread on top and press lightly.
7. Cut in half and serve.
The following is the useful information extracted from the text:
How to Make the Best Pastrami Sandwich at Home:
- Rye bread
- Pastrami
***************************************You can also use it as a function. Here’s a sample program:
from ddgsearch import ddgsearch
import json
query = "how to make a great pastrami sandwich"
# search duckduckgo and scrape the results
results = ddgsearch(query, numresults=2, clean_with_llm=True)
# print the results
print (f"Results: {json.dumps(results, indent=2)}")with the result:
Results: [
{
"url": "https://www.food.com/recipe/hot-pastrami-sandwich-286157",
"title": "Hot Pastrami Sandwich Recipe - Food.com",
"text": "Hot Pastrami Sandwich Recipe - Food.com\n\nRecipes\n\n Breakfast & Brunch Recipes\n \n Lunch Recipes\n \n Appetizers & Snack Recipes\n \n Dinner Recipes\n \n Dessert Recipes\n \n Drink & Cocktail Recipes\n \n Side Dish Recipes\n \n Grilling & BBQ Recipes\n \n Microwave Recipes\n \n Quick & Easy Recipes\n \n Slow-Cooker Recipes\n \n Air Fryer Recipes\n \n Instant Pot Recipes\n \n Baking Recipes\n \n See More\n \nPopular\n\n Trending Now\n \n Casserole Recipes\n \n Chili Recipes\n \n Soup Recipes\n \n Pasta Recipes\n \n Bread Recipes\n \n Cookie Recipes\n \n Salad Recipes\n \n Tofu Recipes\n \n Copycat Recipes\n \n See More\n \nMeat & Seafood\n\n Chicken Recipes\n \n
Salmon Recipes\n \n Pork Chop Recipes\n
\n Ground Beef Recipes\n \n Shrimp Recipes\n \nHealthy & Diet\n\n Keto Recipes\n \n
Healthy Recipes\n \n Vegetarian Recipes\n \n Vegan Recipes\n \n Mediterranean Diet Recipes\n \n Weight Watchers Recipes\n \n
Low-Carb Recipes\n \n Gluten-Free Recipes\n \n See More\n \nHolidays\n\n Dinner Party Recipes\n
\n Game Day Recipes\n \n Valentine's Day Recipes\n \n St. Patrick's Day Recipes\n \n
Easter Recipes\n \n Cinco de Mayo Recipes\n
\n Mother's Day Recipes\n \n Memorial Day Recipes\n \n Juneteenth Recipes\n \n
4th of July Recipes\n \n Halloween Recipes\n
\n Thanksgiving Recipes\n \n Hanukkah Recipes\n \n Christmas Recipes\n \n New Year's Recipes\n \nCuisine\n\n Mexican Recipes\n \n Italian Recipes\n \n Indian Recipes\n
\n Thai Recipes\n \n Korean Recipes\n \n French Recipes\n \n Latin American Recipes\n \n Chinese Recipes\n \n
Japanese Recipes\n \n Spanish Recipes\n
\nSeasonal\n\n Spring Recipes\n \n Summer Recipes\n \n Fall Recipes\n \n
Winter Recipes\n \n See More\n \nicons / profile\n\n Recipes\n \n Lunch/Snacks\n \nHot Pastrami Sandwich\n\n14\n\n Submitted by PalatablePastime\n\n \"A New york deli classic. Some of these sandwiches can be HUGE and are really a marvel (and easily shared).\"\n \nicons / ellipsis / ellipsis-horizontal\n\n save\n \nDownload\n\nPrint\n\nShare\n\nI Made This\n\n photo by PalatablePastime\n \nReady In:\n20mins\n\nIngredients:\n\n 7\n \nServes:\n\n4-8\n\n Nutrition information\n \nAdvertisement\n\ningredients\n\n Units: US\n\n2 \n\n lbs cooked and thinly sliced beef pastrami\n\n4 -8 \n\n slices havarti cheese\n\n coarse grind mustard (optional)\n\n dill pickle slices (optional)\n\n thinly sliced onion (optional)\n\n8 -16 \n\n slices artisan rye bread or 4 -8 sandwich buns\n\n1 \n\n cup beef broth\n\nAdvertisement\n\ndirections\n\nSimmer pastrami in broth until hot.\nDivide meat among bread, using 4 ounces for smaller sandwich, and 8 ounces for large.\nAdd cheese, mustard, pickles, and onions to taste.\nGrill sandwich until toasty and crisp if desired.\nServe warm with chips if desired.\n\nicons / social / pinterest\n\nicons / email\n\nQuestions & Replies\n\n Sign In to Ask a Question \n\n what is the nutritional value for this sandwich\n \nicons / ellipsis / ellipsis-horizontal\n\n Bobbie W.\n \n Reply\n \n See 1 Reply\n \nAdvertisement\n\nReviews\n\nMOST POPULARMOST RECENT\n\nWrite A Review\n\n Great sandwich! I made this on rye bread that I toasted before assembling the sandwich. The pastrami was very tender from simmering in the broth, so the meat just melted in the mouth. I drained the pastrami before putting on the bread so that the sandwich would not be soggy.\n \nicons / ellipsis / ellipsis-horizontal\n\n Ginny Sue\n \n Reply\n \n 4\n \n I moved to central Ohio in 2011. Havent found a place that makes a pastrami sandwich like The Hat in SoCal since I moved here. Looks like I am gonna have to do it myself.\n \nicons / ellipsis / ellipsis-horizontal\n\n
Frank D.\n \n Reply\n \n 2\n \n See 1 Reply\n \n Superb Sandwich, fast and easy with lots of flavor. I added a little mayonnaise to the mustard and grilled the bread with butter on the outside. Served with homemade fries, was a hit. Will make again.\n \nicons / ellipsis / ellipsis-horizontal\n\n Laura L.\n \n Reply\n \n 2\n \n Just a little bland. My construction; Put 3 tbls about 1/4 to 1/3 c. sauerkraut in a small strainer and hold under hot water for 5 sec's. to heat and dilute bitter juice and smell. Squeeze in strainer till near dry. Place and spread on Pumpernickel rye, Marble rye or caraway rye bread. Add a slice of havarti or swiss (keep cheese 1/2\" from edge of bread for melt compensation), spread 1 tsp dijon or coarse mustard on kraut and cheese (substitute 1 or 2 tbls Marie's 1000 island dressing or add to the mustard). Nuke 6 oz. (8 oz. if using pan sized bread) Hillshire Farms pastrami or pre-cooked equivalent for 30 sec., steam or heat in beef broth till hot (DO NOT COOK!). Drip dry and add heated pastrami. Cover with second bread slice. Grill in frying pan. (Very lightly butter outside of top slice. Place in hot pan buttered side down. Lightly butter slice 2 outside while grilling.) Temp and time; roughly 300 F., 5 min a side. Flip if wisp of light smoke is seen and remove when side 2 reveals same smoke. I use the olfactory device in the middle of my face to monitor grilling to a little over done (smoke and light burn smell) for another added great taste variety. Cheese melt migration to bread edge is also a temp indicator to ensure kraut and meat are hot.\n \nicons / ellipsis / ellipsis-horizontal\n\n
D.r. H.\n \n Reply\n \n 1\n \n See 2 Replies\n \n Delicious! Didn't have havarti cheese or rye bread so used pepper jack cheese and grinder rolls, then topped the sandwich with saut\u00e9ed onions and red bellpeppers. Will definitely make this one again.\n \nicons / ellipsis / ellipsis-horizontal\n\n clskinner\n \n Reply\n \nsee 9 more reviews\n\nAdvertisement\n\nTweaks\n\nMOST POPULARMOST RECENT\n\n My own Hot Pastrami sandwich recipe.\n \nicons / ellipsis / ellipsis-horizontal\n\n D.r. H.\n \n Reply\n \n wow this is a wonderful sandwich with many possibilites, I used corned beef in place of pastrami and provolone cheese, made as stated this was wonderful, I strongly suggest to reduce the broth by half which is what I did otherwise I think it would make the bread too wet, thanks for sharing Lori!\n \n Kittencalrecipezazz\n \n Reply\n \nRECIPE SUBMITTED BY\n\nPalatablePastime\n\n United States\n \nicons / following\n\n Follow Me\n \n406\n Followers\n \n1414\n Recipes\n \n42\n Tweaks\n \n You know me as Sue L or Sue Lau. I write a food blog at palatablepastime.com which specializes in Midwestern, Amish, Southern, and Ethnic recipes. Most of which are my own recipes. There are a few there that aren't, but not many. And really, since the best of my recipes here are my recipes, it tells me you guys like me just a teensy bit. ;)\nI post all my latest stuff on my blog including new recipes, updates to older recipes and (hopefully) much better photos. \nSo don't be shy in dropping by.\nIf you like my stuff here you will LOVE my stuff there. Seriously.\n \nView Full Profile\n\nAdvertisement\n\nAdvertisement\n\nAdvertisement\n\nYOU'LL ALSO LOVE\n\n
28 Best New York Recipes\n \n28 photos\n \n Hot Pastrami Baked Sandwiches\n \n by\n Chef Jean\n\n7\n\n How to Make Meatball Sub Kolaches\n \n 28 Springtime Salads for Easter\n
\n View All Recipes\n \nRelated Pages\n\n Simple Hot Pastrami and Swiss Sandwiches\n
\n Beer Pastrami Sandwich\n \n Katz's Corned Beef and Pastrami Sandwich\n \n Baked Turkey-Pastrami Sandwich\n \n Hot Pastrami Baked Sandwiches\n \n Pastrami Chili Cheese Fries\n \n Hot Pastrami Wrap\n \n Pastrami & Red Cabbage Sandwich\n \n Mini Pastrami Packages\n \nFind More Recipes\n\nSearch\n\nBack To Top\n\nicons / social / facebook\n\nicons / social / pinterest\n\nicons / social / twitter\n\nicons / social / youtube\n\nicons / email\n\n All Categories\n \n
Site Map\n \n About Us\n \n Help\n \n The Discovery Family of Networks\n \nHGTV\n\nDIY Network\n\nFood Network\n\nCooking Channel\n\nTravel Channel\n\nHGTV Poland\n\n \u00a9 2023 Warner Bros. Discovery, Inc. or its subsidiaries and affiliates. All rights reserved.\n \n Advertise\n \n AdChoices\n \n Privacy Notice\n \n Visitor Agreement\n \n California Privacy Notice\n \n Do Not Sell or Share My Personal Information\n ",
"useful_text": "Hot Pastrami Sandwich Recipe: This recipe is a classic deli sandwich made with pastrami, Swiss cheese, and a tangy mustard. It's easy to make and perfect for a quick lunch or dinner. Serve with a side of chips or a salad for a complete meal.\nHot Pastrami Sandwich Recipe: This recipe is a classic sandwich made with pastrami, Swiss cheese, and mustard on rye bread. It's easy to make and great for a quick lunch or dinner. The air fryer or Instant Pot can be used to make this sandwich in minutes.\nHot Pastrami Sandwich Recipe: This recipe is a classic sandwich that is easy to make and packed with flavor. It features thinly sliced pastrami, melted Swiss cheese, and a tangy mustard sauce on a toasted bun. It's a great lunch or dinner option that can be made in minutes.\nThis recipe is for a Hot Pastrami Sandwich. It is made with rye bread, pastrami, Swiss cheese, and a special sauce. The sauce is made with mayonnaise, Dijon mustard, and pickle relish. The sandwich is served with lettuce and tomato. It is a great lunch or dinner option for anyone looking for a tasty sandwich.\nThe useful information from the text is that the webpage contains recipes for Cinco de Mayo, Mother's Day, Memorial Day, Juneteenth, 4th of July, Halloween, Thanksgiving, Hanukkah, Christmas, and New Year's. It also contains recipes for Mexican, Italian, and Indian cuisines.\nHot Pastrami Sandwich Recipe: This classic sandwich is made with thinly sliced pastrami, Swiss cheese, and a tangy mustard spread on toasted rye bread. It's a great way to enjoy a delicious lunch or dinner.\nIngredients:\n- Cooked and thinly sliced beef pastrami\n- Slices of Havarti cheese\n- Coarse grind mustard (optional)\n- Dill pickle slices (optional)\n- Thinly sliced onion (optional)\n- Slices of artisan rye bread or sandwich buns\n- Beef broth\n\nDirections:\n1. Preheat oven to 350\u00b0F.\n2. Place pastrami slices on a baking sheet.\n3. Top with cheese slices.\n4. Bake in preheated oven for 10 minutes.\n5. Spread mustard on bread slices or buns.\n6. Top with pastrami and cheese.\n7. Add pickles, onions, and any other desired toppings.\n8. Place sandwiches on a baking sheet.\n9. Pour beef broth over sandwiches.\n10. Bake in preheated oven for 10 minutes.\n11. Serve hot.\nHot Pastrami Sandwich Recipe:\n- Simmer pastrami in beef broth until hot\n- Divide meat among bread, using 4 ounces for smaller sandwich, and 8 ounces for large\n- Add cheese, mustard, pickles, and onions to taste\n- Grill sandwich until toasty and crisp if desired\n- Serve warm with chips if desired\n- Reviewer Ginny Sue recommends toasting the rye bread before assembling the sandwich and draining the pastrami before putting on the bread so that the sandwich would not be soggy.\nThis Hot Pastrami Sandwich Recipe is fast and easy to make with lots of flavor. It can be served with homemade fries for a delicious meal. To add more flavor, you can add a little mayonnaise to the mustard and grill the bread with butter on the outside. To reduce the bitterness of the sauerkraut, put 3 tablespoons of it in a small strainer and hold under hot water for 5 seconds. Squeeze the strainer to remove the bitter juice and smell.\nHot Pastrami Sandwich Recipe:\n- Heat and dilute bitter juice and smell of pastrami by holding under hot water for 5 seconds and squeezing in a strainer till near dry.\n- Place and spread on Pumpernickel rye, Marble rye or caraway rye bread.\n- Add a slice of havarti or swiss cheese (keep cheese 1/2\" from edge of bread for melt compensation).\n- Spread 1 tsp dijon or coarse mustard on kraut and cheese (substitute 1 or 2 tbls Marie's 1000 island dressing or add to the mustard).\n- Nuke 6 oz. (8 oz. if using pan sized bread) Hillshire Farms pastrami or pre-cooked equivalent for 30 sec., steam or heat in beef broth till hot (DO NOT COOK!). Drip dry and add heated pastrami.\n- Cover with second bread slice.\n- Grill in frying pan (very lightly butter outside of top slice. Place in hot pan buttered side down. Lightly butter slice 2 outside while grilling).\n- Temperature and time: roughly 300 F., 5 min a side. Flip if wisp of light smoke is seen and remove when side 2 reveals same smoke.\nMy own Hot Pastrami sandwich recipe.\nDelicious! Didn't have havarti cheese or rye bread so used pepper jack cheese and grinder rolls, then topped the sandwich with saut\u00e9ed onions and red bellpeppers. Will definitely make this one again.\nSuggestion to reduce the broth by half.\nUseful Information:\n- Hot Pastrami Sandwich Recipe \n- Submitted by PalatablePastime\n- Suggestion to reduce broth by half to avoid making the bread too wet\n- PalatablePastime specializes in Midwestern, Amish, Southern, and Ethnic recipes\nThe useful information from the text is:\n\nHot Pastrami Sandwich Recipe by Chef Jean. \nSimple Hot Pastrami and Swiss Sandwiches, Beer Pastrami Sandwich, Katz's Corned Beef and Pastrami Sandwich, Baked Turkey-Pastrami Sandwich, and Hot Pastrami Baked Sandwiches.\nHot Pastrami Baked Sandwiches\nPastrami Chili Cheese Fries\nHot Pastrami Wrap\nPastrami & Red Cabbage Sandwich\nMini Pastrami Packages"
},
{
"url": "https://pocketchangegourmet.com/pastrami-sandwich/",
"title": "How to Make a Classic Pastrami Sandwich",
"text": "How to Make a Classic Pastrami Sandwich\n\nPocket Change GourmetBudget Friendly Meals with a Little FlairFollow Us\n\nAll Recipes\n\nFeatured\nQuick & Easy Meals\nAppetizers & Starters\nSalads & Sides\nMain Dishes\n\nMenu Planning\nSeasonal\n\nFrom the Grill\nFrom the Garden\nComfort Food\nHolidays\n\nEssentials\n\n5 Ingredients or Less\nKitchen Tips\nHomemade Pantry\nVideo Tutorials\n\n Best Pastrami Sandwich Recipe\nJune 28, 2022 by Eliot Martin 1 Comment \nThe Pastrami sandwich as we know it today comes from pastrami on rye\u2014a classic sandwich from the bustling streets of 20th-century New York. Pastrami on rye continued to be developed throughout New York\u2014and particularly within American Jewish communities\u2014and has since become a popular menu item for many enthusiasts of kosher meat and sandwiches lovers alike.\nMaking the sandwich on your weekly menu is far more involved than placing meat on bread. We\u2019ll be going over a detailed recipe for a killer pastrami sandwich, talking a little bit about variations on pastrami itself, and hopefully providing your guests with a great meal.\n\nImage: CC by 3.0, Erika39, via Wikipedia Commons\n\nHow to Make the Best Pastrami Sandwich at Home\n\nImage: CC by A-SA 4.0, Xisler, via Wikipedia Commons\nWhat You Need to Make a Pastrami Sandwich\nRye breadPastrami TomatoLettuceOnionCheeseButterSalt and pepper (to taste)\nHow to Make the Perfect Pastrami Sandwich\nSince there are tons of ways to build upon the simplistic nature of a pastrami sandwich, we\u2019ll be going over the basics and touch upon a few variations you can try out a little later on.\n\nImage: CC by A-SA 2.0, star5112, via Flickr\n1. Side Dishes for Your Pastrami Sandwich\nWhile it might seem counterintuitive, to start our pastrami on rye, we\u2019re going to need to get the onions going first. \nAfter slicing up enough farmer\u2019s market onions to top off the number of sandwiches you need, prepare a pan by dropping in a good amount of butter and melting over medium heat. Once we\u2019ve got a heated pan and cooked butter, drop in your onions and begin the caramelization process.\nCaramelization can take several minutes, so we\u2019re going to come back to these once they\u2019ve turned a nice golden brown.\nWhile we\u2019re waiting for those to cook, it is time to get back to the rest of our ingredients. \nUsing a cutting board, slice up your tomatoes\u2014leaving enough thickness to give a nice height and flavor to the finished sandwich. If you need to prepare your lettuce, this is also the best time to get on that. \nDon\u2019t forget to go back over and stir the onions on occasion to prevent uneven cooking.\n2. The Bread\n\nImage: CC 0 by Public Domain, anaterate, via Pixabay\nNext, we\u2019ll need to work on our bread. For the sake of tradition, consider using rye bread or a similar type\u2014but whatever you choose, make sure you use a thin (but not too thin) slice to bring out the other flavors in the sandwich. Other options to consider include sourdough and strong whole wheat bread. \nWe\u2019re going to want to get these toasted, which can be done in two ways\u2014either by utilizing your oven, or preferably coating with butter and tossing on the grill. If either option isn\u2019t possible for your kitchen, wait until after your onions have caramelized and use the empty pan to get started on toasting. \nBe sure to go easy on the butter\u2014we\u2019re going for toasted bread, not Texas Toast or garlic bread.\n3. The Meat\nIt is time to get back to the pastrami, but how you\u2019ve purchased the meat plays a major factor in how you serve your sandwich. Purchased sliced pastrami from your local deli means there is no need to cook the meat. Simply lay on the bottom slice atop your lettuce and tomato, slide the caramelized onions on top, and you\u2019re good to go.\nBringing out the smokey flavor of the pastrami requires more steps to go over.\nAdding some melted goat cheese on top and stacking your pastrami slices in the oven comes next.\u00a0 Top it up with cheese, and blast it until melted. This will give you a warmer and gooey taste and works best for those who prefer a hotter sandwich. \nGetting classy with your sandwich suggests cooking your pastrami much like your onions. Using a pan coated in olive oil add, in about half of the pastrami you\u2019re planning to use and cook for about a half-minute per side. \nDo the same for the other half and keep cooking until we get a similar caramelized appearance as we did on our onions. Once you\u2019re done, add the pastrami to the base we\u2019ve already made and salt and pepper to taste.\n4. The Pastrami Sandwich\n\nImage: CC by A 2.0, jeffreyw, via Wikipedia Commons\nChoosing to cook your pastrami, try finishing it off by giving the entire assembled sandwich time to toast on the pan. If you need to, use your spatula to press down on the bread to keep a nice, even coat. Let the sandwich press do the rest for you.\nOnce you\u2019re done, serve immediately or slice in half. For all the hard work you\u2019ve put into this sandwich, it\u2019s understandable to want to impress guests with a killer cross-section. \nFor an extra splash of tradition, offer deli mustard and make sure your pastrami is stacked too high for comfort. \nWith this dish, there\u2019s no such thing as too much meat. \nMelt butter in a pan and begin caramelizing your onions.Prepare thick slices of tomato and thin slices of toasted bread.Cook (or neglect to cook) your pastrami and assemble immediately.Toast generously on both sides, cut in half, and enjoy!.\nVariations and Meal Ideas for Pastrami Sandwich\nA sandwich as simple as pastrami on rye is practically begging for innovation\u2014and we\u2019ve got several options available to you to try and breathe new life into this old classic. We\u2019ll be going over these options, other dishes you can try to serve alongside the pastrami and rye, and a brief look into the importance of the meat itself. \nUsing\u00a0The Right Pastrami\n\nImage: CC by A 2.0, Arnold Gatilao, via Wikipedia Commons\nChoosing whether or not to use pre-sliced pastrami or cooking the pastrami yourself has a lot to do with the history of the meat and the quality that you\u2019ve purchased.\nPastrami is usually well seasoned and spiced, brined, smoked, and steamed. The Jewish roots mean that a traditional pastrami sandwich comes from the kosher beef plate. The same goes for the exhaustive nature of making pastrami. Although, in the modern-day, one opts to find easier pastrami preparation from beef brisket.\u00a0\nGoing for a traditional pastrami sandwich implies being willing to go the extra mile.\u00a0 Choosing the pastrami made from the beef plate provides a proven base from which to cook the meat yourself. However, if you want to keep things simple and easy to make, stick with pre-sliced pastrami. Once accented correctly with the right ingredients results in a great sandwich for all.\nIt all depends on what you\u2019re going for, the kind of tastes you want to bring out, and your price range. \nShaking Things Up\n\nImage: CC 0 by Public Domain, (Unidentified Author), via Max Pixel\nSide items remain an important accent for your pastrami sandwich. Remember that the limit appears not among the bare bones of a pastrami on rye.\nLike other sandwiches commonly served at delis, experimenting with various cheeses to meet certain tastes makes a better meal. Throwing\u00a0a fried egg to add a little more protein and thickness surely gives a more irresistible taste. Avocado stands as a popular choice in modern delis. This adds more green in balancing out all that red meat.\nIf you\u2019re looking to shake things up, try adding red or green peppers to bring out the natural flavors of pastrami. Deli-style neutral pasta like macaroni incorporates the full meal into a small sandwich. Whatever you choose to do, try to keep your big changes to one solid deviation. A good pastrami sandwich denotes the lack of distraction. Trying too hard usually brings out diminishing returns. \nSoups, Salads, and Side items\n\nImage: CC by A-SA 3.0, Jonathunder, via Wikipedia Commons\nAs a simple-but-filling sandwich, the pastrami on rye can be served in multiple portions to constitute a full meal.\u00a0 Pairing with other items keeps things interesting. Here are a few ways you can tie together a full meal with the pastrami on rye, without breaking the bank:\nMost summer salads pair great with pastrami! Try adding a Cobb or Mediterranean salad that gives you a bit more garden with this meat-heavy item. Tomato-based soups also work well with pastrami. Moreover, hungry guests satisfied with the dipping sauce guarantees more cravings for the meal.\nOther side items include classic options like chips or onion rings. But anyone looking to serve a pastrami sandwich that includes avocado might prefer lighter sides like coleslaw. \nIf you want to keep with the history of the sandwich, go with a kosher pickle wedge. It sounds relatively inexpensive yet works great with the sandwich. It shows a little bit of understanding of where your meal came from. \nHow Do You Like to Have Your Pastrami Sandwich?\nThe pastrami sandwich holds a great option for a simple lunch option or a quick and tasty dinner item. Traditionally cooked over a grill or purchased pre-made makes a good choice! Mastering the art of such a simple sandwich gives you another set of gourmet skills. Choosing to cook your own works out as more beneficial and rewarding than it initially appears. \nBe sure to try out the pastrami sandwich with Mediterranean meals, in conjunction with other finger foods, or for a lazy night around the house. \u00a0\n\nImage: CC by Jeremy Keith, PACAF, via Flickr\nFeatured Image: CC by 3.0, Erika39, via Wikipedia Commons.\n\nFacebook\n\nTwitter\n\nGoogle+\n\nPinterest\n\nEliot Martin\u00a0Eliot Martin is a book lover and a public relations graduate. He currently lives in the Phoenix area, where he spends most of his time playing video games or reading a good book. He enjoys going to comic cons and spending time with his friends and family, and when he is not writing for his own cooking blog, he keeps our readers hooked with his articles.\nCategories: Sandwiches Find More: American Jewish sandwich, Cobb or Mediterranean salad, New York sandwich, pastrami, pastrami sandwich, Recipes, Rye bread, sandwichComments\n\n Jimmy t\n\n\t\t\t\tsays\t\t\t\t\n\nJanuary 6, 2022 at 11:32 am\n\nI absolutely love rueben sandwiches. Pastrami sandwiches are quick and easy way to ease the crave for the rueben. Mine is really easy and quick. Lightly toast seeded Jewish rye bread and and sauce to both. This can be anything from mustard to kicked up russian dressing. Gently warm the pastrami with cheese on top, just until the cheese stars to melt. Put it together and you are done. Of course, when I have time, I will ad sauerkraut etc,\u2026but usually for a quick fake Reuben\u2026.it is just cheese and mustard.\n\nReply \n\nSpeak Your Mind Cancel replyName *\nEmail *\nWebsite\n \n\u0394 \n As a participant in the Amazon Services LLC Associates Program, this site may earn from qualifying purchases. We may also earn commissions on purchases from other retail websites.\n\nSearch Recipes\n\nCategories\nCategories\nSelect Category\n5 Ingredients or Less\nAll Recipes\nAppetizers & Starters\nBBQ & Grilling\nBreakfast & Brunch\nComfort Food\nDecorating Tips\nDrinks\nEssentials\nEthnic\nFan Favorites\nFeatured\nFrom the Garden\nFrom the Grill\nGiveaway\nHealthy\nHolidays\nHome Page \u2013 Slider\nHomemade Pantry\nKid Friendly\nKitchen Tips\nMain Dishes\nMake Ahead Meals\nMeat\nMenu Planning\nPasta\npocketchangegourmet\nPoultry\nQuick & Easy Meals\nSalads & Sides\nSandwiches\nSeasonal\nSlow Cooker\nSnacks\nSoups, Stews and Casseroles\nSponsored Post\nUncategorized\nVideo Tutorials\n\nReturn to top of pageCopyright \u00a9\u00a02023 \u00b7 Genesis Framework \u00b7 WordPress \u00b7 Log in\n\nPin It on Pinterest\n\nShare This\n\nFacebook\n\nTwitter\n\nGoogle+\n\nPinterest\n\nAbout - Contact - Privacy Policy\n",
"useful_text": "Best Pastrami Sandwich Recipe\nIngredients:\n- Rye bread\n- Pastrami\n- Mustard\n- Pickles\n\nInstructions:\n1. Preheat oven to 350\u00b0F.\n2. Place rye bread slices on a baking sheet and toast in the oven for 5 minutes.\n3. Place pastrami on the toasted rye bread slices.\n4. Spread mustard on the pastrami.\n5. Top with pickles.\n6. Serve.\nThe following is the useful information extracted from the text:\n\nTo make a perfect pastrami sandwich, you will need rye bread, pastrami, tomato, lettuce, onion, cheese, butter, salt and pepper. Begin by caramelizing the onions in a pan with butter over medium heat. Serve the sandwich with side dishes such as coleslaw, potato salad, or pickles.\nTo make a classic pastrami sandwich, you'll need rye bread, tomatoes, lettuce, onions, and pastrami. Caramelize the onions by cooking them over low heat until they turn a nice golden brown. Slice the tomatoes and prepare the lettuce. Toast the bread using either an oven or by coating with butter and grilling. Assemble the sandwich by layering the pastrami, caramelized onions, tomatoes, and lettuce. Enjoy!\nThe text describes how to make a classic pastrami sandwich. It explains that you can either use your oven or grill to toast the bread. It also explains that if you purchased sliced pastrami from the deli, you can simply lay it on the bottom slice of bread with lettuce and tomato. It also suggests adding melted goat cheese and stacking the pastrami slices in the oven to bring out the smoky flavor. Finally, it suggests cooking the pastrami like you would a steak.\nThis classic pastrami sandwich is made by cooking the pastrami in a pan with olive oil for about a half-minute per side until it is caramelized. The sandwich is then toasted on the pan and served with deli mustard. The pastrami should be stacked too high for comfort.\nPastrami is usually well seasoned and spiced, brined, smoked, and steamed. It is served on thick slices of toasted bread with deli mustard and caramelized onions. Variations and meal ideas for the pastrami sandwich include adding different condiments, such as cheese, pickles, and coleslaw. Other dishes to serve alongside the sandwich include potato salad, coleslaw, and pickles. When choosing pastrami, it is important to consider the quality of the meat and whether to use pre-sliced or cook the pastrami yourself.\nA traditional pastrami sandwich is made from kosher beef plate, but pre-sliced pastrami can also be used. Side items such as pickles, mustard, and coleslaw can be added to the sandwich to enhance the flavor. Other deli sandwiches can also be served with pastrami.\nThe pastrami sandwich is a classic deli sandwich that can be served with various cheeses, fried eggs, avocado, red or green peppers, and macaroni. To keep the sandwich simple and flavorful, it is best to limit the changes to one deviation. To make a full meal, the sandwich can be paired with other items such as soups, salads, and side dishes.\nThe pastrami sandwich is a classic lunch or dinner option. It can be cooked over a grill or purchased pre-made. To pair with the sandwich, try adding a Cobb or Mediterranean salad, tomato-based soups, chips, onion rings, coleslaw, or a kosher pickle wedge.\nThe pastrami sandwich is a classic American Jewish sandwich made with rye bread, pastrami, and other ingredients. It can be cooked over a grill or purchased pre-made. It is a great accompaniment to Mediterranean meals, finger foods, or for a lazy night at home.\nThe text contains a recipe for a classic pastrami sandwich. The recipe is as follows:\n\n- Lightly toast seeded Jewish rye bread and spread sauce on both sides.\n- Gently warm the pastrami with cheese on top, just until the cheese starts to melt.\n- Put it together and you are done.\n- For a more elaborate sandwich, add sauerkraut, Russian dressing, or other toppings."
}
]Here’s the actual code:
So how does it work?
1: Call DuckDuckGo
First, it calls duckduckgo to do a web search. Why duckduckgo? Because they are the easiest to use. No API key is required, just do the following:
from duckduckgo_search import ddg
...
query = "how to make a great pastrami sandwich"
results = ddg(query, max_results=10)I’m using the python package duckduckgo_search.
Easy! The results are a list of dictionaries as follows:
[
{
"title": "How to Make a Classic Pastrami Sandwich - Pocket Change Gourmet",
"href": "https://pocketchangegourmet.com/pastrami-sandwich/",
"body": "Using a cutting board, slice up your tomatoes\u2014leaving enough thickness to give a nice height and flavor to the finished sandwich. If you need to prepare your lettuce, this is also the best time to get on that. Dont forget to go back over and stir the onions on occasion to prevent uneven cooking. 2. The Bread."
},
{
"title": "Ultimate Pastrami Sandwiches Recipe: How to Make It - Taste of Home",
"href": "https://www.tasteofhome.com/recipes/ultimate-pastrami-sandwiches/",
"body": "Drain cabbage. In a small bowl, combine the salt, celery seed, pepper and remaining sugar and vinegar; pour over cabbage and toss to coat. On an ungreased baking sheet, divide pastrami into 4 stacks; top each with cheese. Bake at 450\u00b0 for 2-3 minutes or until cheese is melted. Place pastrami on 4 toast slices."
}
]I’m currently ignoring everything except the urls, like this:
urls = [result['href'] for result in results]
urls = urls[:numresults] # just in case2: Use the Scrapy webspider to download all the pages
Scrapy is a serious application for spidering through websites.
Scrapy is available via the python package scrapy.
It’s intended to be used directly on the cli, but you can actually use it inside your python scripts:
That’s how I’m using it.
To make use of Scrapy, I need to create a scrapy.Spider subclass:
import scrapy
class MySpider(scrapy.Spider):
'''
This is the spider that will be used to crawl the webpages.
We give this to the scrapy crawler.
'''
name = 'myspider'
start_urls = None
clean_with_llm = False
results = []
def __init__(self, start_urls, clean_with_llm, *args, **kwargs):
super(MySpider, self).__init__(*args, **kwargs)
self.start_urls = start_urls
self.clean_with_llm = clean_with_llm
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(url, callback=self.parse)
def parse(self, response):
# In here, response contains the downloaded web page.
# I'll get to what to actually do with it below
...
It’s pretty simple: I give it a list of urls to visit, and the clean_with_llm setting. In start_requests, I yield each url, which tells Scrapy to go fetch them. In parse, scrapy is giving me one of the web pages, and I can process it.
Note: We could use selenium, or even better playwright, to do a better job of fetching the page, where it would run client side javascript first. I’ve not done this in this article, but note that it is possible.
Now that we’ve got the spider class, how do we invoke scrapy? Like this:
from scrapy.crawler import CrawlerProcess
process = CrawlerProcess()
process.crawl(MySpider, urls, clean_with_llm)
process.start()
# here the spider has finished downloading the pages and cleaning them up
return MySpider.resultsThere are apparently a whole ton of settings you can use for scrapy, I’ve just used it with all defaults. Seems to work, but it’s worth investigating them for more complex tasks.
Note: I do all kinds of hackery to control logging in ddgsearch, and I’ve left that out of this article. Getting scrapy to do its job without barfing into stdout isn’t simple, have a look at my code to behold my mad hacks.
So the final result from the function is in MySpider.results. But how does that get set? It gets set when we…
3. Use the parse method to clean up and return the results
Here’s my parse method:
def parse(self, response):
body_html = response.body.decode('utf-8')
url = response.url
soup = bs4.BeautifulSoup(body_html, 'html.parser')
title = soup.title.string
text = soup.get_text()
text = remove_duplicate_empty_lines(text)
if self.clean_with_llm:
useful_text = extract_useful_information(url, title, text, 50)
else:
useful_text = readability(body_html)
useful_text = remove_duplicate_empty_lines(useful_text)
self.results.append({
'url': url,
'title': title,
'text': text,
'useful_text': useful_text
})What’s this doing?
First, it gets the body from the response. That’s the actual web page’s html.
hmm, looking at that, maybe I need to check the status code before proceeding? My experience is that scrapy doesn’t call parse unless the response is a 200.
Now, I need to return url, title, text, and useful_text.
url comes from response, easy.
I use BeautifulSoup to parse the web page, and get the title and the text. Also fairly easy.
Unfortunately, text can be long and full of garbage. I just want the useful bits that are about what the search is about. How to do this?
Method 1: readability
If we don’t set clean_with_llm, then the code uses a library called readability-lxml to clean up the page. It traverses the DOM, is full of hackery, and returns you a “readable” version of the page. It’s based on readability.js:
Thanks to my friend Lord Xelon for pointing me to readability.js
This is pretty good, but I noticed that it can throw away large sections of useful information. It is a bunch of hackery in the end.
Benefits: It’s fast, it’s free (no proprietary API calls)!
Drawbacks: It can throw away useful information.
Method 2: Use text-davinci-003
If you set clean_with_llm, we call GPT. This approach really distils out the useful text from a messy web page. I use the following prompt and send it to OpenAI’s text-davinci-003 to complete:
prompt = f"""
Here is a url: {url}
Here is its title: {title}
Here is some text extracted from the webpage by bs4:
---------
{text}
---------
Web pages can have a lot of useless junk in them.
For example, there might be a lot of ads, or a
lot of navigation links, or a lot of text that
is not relevant to the topic of the page. We want
to extract only the useful information from the text.
You can use the url and title to help you understand
the context of the text.
Please extract only the useful information from the text.
Try not to rewrite the text, but instead extract
only the useful information from the text.
"""Of course, the text from the web page can be too long for text-davinci-003, so the code breaks it into smaller pieces, sends several prompts in parallel, then concatenates them into one result. The code to do this is a bit involved, but not too bad, have a look at the source if you want to know more.
The results are pretty good, but it does rewrite the page’s text, it can introduce subtle errors. The prompt is a work in progress.
Benefits: It does a pretty great job of extracting useful info (although reworded and maybe introducing subtle errors).
Drawbacks: It’s expensive and slow.
Conclusion
This utility/function isn’t all that useful on its own, it’s more of a tech demonstration. The point of this is to have a way to turn web searches into a list of salient content. Why do I want to do this? So I can chop the results into chunks, calculate embeddings, and put them in a vector database.
And why do I want to do that? So that I can look these chunks up by similarity to a user’s question, and stuff them into a prompt for GPT to answer. But that’s for next time.
