by
Braxx
• Contributor II
- 4509 Views
- 6 replies
- 2 kudos
I am trying to group by a data frame by "PRODUCT", "MARKET" and aggregate the rest ones specified in col_list. There are much more column in the list but for simplification lets take the example below.Unfortunatelly I am getting the error:"TypeError:...
- 4509 Views
- 6 replies
- 2 kudos
Latest Reply
The error you're encountering, "TypeError: unhashable type: 'Column'," is likely due to the way you're defining exprs. In Python, sets use curly braces {}, but they require their items to be hashable. Since the result of sum(x).alias(x) is not hashab...
5 More Replies
- 4062 Views
- 7 replies
- 4 kudos
When saving notebook to GiHub repo, it is stripped to Python source code. Is it possible to save it in the ipynb formt?
- 4062 Views
- 7 replies
- 4 kudos
Latest Reply
When I save+commit+push my .ipynb file to my linked git repo, I noticed that only the cell inputs are saved, not the output. This differs from the .ipynb file I get when I choose "File / Export / iPython Notebook". Is there a way to save the cell o...
6 More Replies
by
MCosta
• New Contributor III
- 6902 Views
- 11 replies
- 20 kudos
Hi ML folks,
We are using Databricks to train deep learning models. The code, however, has a complex structure of classes. This would work fine in a perfect bug-free world like Alice in Wonderland.
Debugging in Databricks is awkward. We ended up do...
- 6902 Views
- 11 replies
- 20 kudos
Latest Reply
Has this been solved yet; a mature way to debug code on databricks. I'm running in the same kind of issue.Variable explorer can be used and pdb, but not the same really..
10 More Replies
- 12298 Views
- 3 replies
- 4 kudos
Hello, I am doing the Data Science and Machine Learning course.
The Boston housing has unintuitive column names. I want to rename them, e.g. so 'zn' becomes 'Zoning'.
When I run this command:
df_bostonLegible = df_boston.rename({'zn':'Zoning'}, axi...
- 12298 Views
- 3 replies
- 4 kudos
Latest Reply
If df_boston is a DataFrame, but you still face issues, try an alternative syntax: df_boston = df_boston.rename(columns={'zn': 'Zoning'}).Make sure df_boston is a proper DataFrame and you're using a recent version of Pandas.
2 More Replies
- 1236 Views
- 3 replies
- 1 kudos
Hi,I have created a python wheel with the following code. And the package name is rule_engine"""The entry point of the Python Wheel"""import sysfrom pyspark.sql.functions import expr, coldef get_rules(tag): """ loads data quality rules from a table ...
- 1236 Views
- 3 replies
- 1 kudos
Latest Reply
You can find more details and examples here https://docs.databricks.com/en/workflows/jobs/how-to/use-python-wheels-in-workflows.html#use-a-python-wheel-in-a-databricks-job
2 More Replies
- 1635 Views
- 9 replies
- 3 kudos
Databricks Certified Associate Developer for Apache Spark 3.0
- 1635 Views
- 9 replies
- 3 kudos
Latest Reply
Hey I am looking for sample papers for the above exam other than the one provided by databricks do any one have any idea about it
8 More Replies
- 1977 Views
- 7 replies
- 1 kudos
Hi Everyone,I'm planning to use databricks python cli "install_libraries"can some one pls post examples on function install_libraries https://github.com/databricks/databricks-cli/blob/main/databricks_cli/libraries/api.py
- 1977 Views
- 7 replies
- 1 kudos
Latest Reply
Here you go using Python SDKfrom databricks.sdk import WorkspaceClientfrom databricks.sdk.service import computew = WorkspaceClient(host="yourhost", token="yourtoken")# Create an array of Library objects to be installedlibraries_to_install = [compute...
6 More Replies
- 1912 Views
- 4 replies
- 0 kudos
Hello,I 'm trying to execute databricks notebook form a python source code but getting error.source code below------------------from databricks_api import DatabricksAPI
# Create a Databricks API client
api = DatabricksAPI(host='databrick_host', tok...
- 1912 Views
- 4 replies
- 0 kudos
Latest Reply
The error you are encountering indicates that there is an issue with establishing a connection to the Databricks host specified in your code. Specifically, the error message "getaddrinfo failed" suggests that the hostname or IP address you provided f...
3 More Replies
by
T_1
• New Contributor III
- 15394 Views
- 13 replies
- 3 kudos
Trying to use displayHTML from w/in a Python module gets a Python exception:NameError: name 'displayHTML' is not definedand I've found no way around this. It seems to be something at the UI layer or something, not a Python function that can be refere...
- 15394 Views
- 13 replies
- 3 kudos
Latest Reply
Holy Guacamole Batman! It works finally!!!! Wow, thanks @ptweir That's awesome! I can go back and update my doc (and code, to just use databricks the same, now, and Jupyter!) and it'll work by default. It's great they fixed it, shame they never told ...
12 More Replies
by
Braxx
• Contributor II
- 8387 Views
- 3 replies
- 1 kudos
Let's say I want to check if a condition is false then stop the execution of the rest of the script. I tried with two approaches:1) raising exceptionif not data_input_cols.issubset(data.columns):
raise Exception("Missing column or column's name mis...
- 8387 Views
- 3 replies
- 1 kudos
Latest Reply
In Jupyter notebooks or similar environments, you can stop the execution of a notebook at a specific cell by raising an exception. However, you need to handle the exception properly to ensure the execution stops. The issue you're encountering could b...
2 More Replies
- 1975 Views
- 4 replies
- 2 kudos
I have a function for rotating images written in python:from PIL import Image
def rotate_image(image, rotation_angle):
im = Image.open(image)
out = im.rotate(rotation_angle, expand = True)
return outI now want to use this function as a pyspark ...
- 1975 Views
- 4 replies
- 2 kudos
Latest Reply
Stock photos, I've come to realize, are the catalysts of imagination. This website's vast reservoir of images new york seal sparks ideas that ripple through my projects. They empower me to envision the previously unimagined, helping me breathe life i...
3 More Replies
- 4652 Views
- 9 replies
- 3 kudos
yesterday all of my notebooks seemingly changed to have python formatting (which seems to be in this week's release), but the unintended consequence is that shift + tab (which used to show docstrings in python) now just un-indents code, and tab inser...
- 4652 Views
- 9 replies
- 3 kudos
- 9445 Views
- 15 replies
- 7 kudos
Hi,​Let's assume I have these things:Binary column containing protobuf-serialized dataThe .proto file including message definition​What different approaches have Databricks users chosen to deserialize the data? Python is the programming language that...
- 9445 Views
- 15 replies
- 7 kudos
Latest Reply
We've now added a native connector with parsing directly with Spark Dataframes. https://docs.databricks.com/en/structured-streaming/protocol-buffers.htmlfrom pyspark.sql.protobuf.functions import to_protobuf, from_protobuf
schema_registry_options = ...
14 More Replies
- 40383 Views
- 14 replies
- 7 kudos
I have a python notebook A in Azure Databricks having import statement as below:
import xyz, datetime,...
I have another notebook xyz being imported in notebook A as shown in above code. When I run notebook A, it throws the following error:
ImportEr...
- 40383 Views
- 14 replies
- 7 kudos
Latest Reply
Create a repository containing an __init__.py fileAdd your library as .py file(s). Let's imagine that our library is composed by multiple sub-folders consolidated in "my_folder", one of sub-folders is named as "math_library" and contains my_awesome_l...
13 More Replies