- 4611 Views
- 4 replies
- 4 kudos
Using ipywidgets latest versions
Hello everyone,I upgraded my cluster to DBR 13.0, which comes with ipywidgets version 7.7.2 installed.However, I want to use the TagsInput widget, which is new since version 8.0.4.If i upgrade the ipywidgets package to version 8.0.4, none of the widg...
- 4611 Views
- 4 replies
- 4 kudos
- 4 kudos
I can confirm that installing a newer ipywidgets library version at a cluster level does not resolve these issues. The arcgis library relies on ipywidgets v8 to render maps. Even when I install ipywidgets > 8 at the cluster level, the widgets still d...
- 4 kudos
- 775 Views
- 3 replies
- 1 kudos
- 775 Views
- 3 replies
- 1 kudos
- 1 kudos
Unfortunately as this is from the Azure side I do not have specific steps or example i can share
- 1 kudos
- 30844 Views
- 19 replies
- 15 kudos
How to enable/verify cloud fetch from PowerBI
I tried to benchmark the Powerbi Databricks connector vs the powerbi Delta Lake reader on a dataset of 2.15million rows. I found that the delta lake reader used 20 seconds, while importing through the SQL compute endpoint took ~75 seconds. When I loo...
- 30844 Views
- 19 replies
- 15 kudos
- 15 kudos
I'm troubleshooting slow speeds (~6Mbps) from Azure Databricks to the PowerBI Service (Fabric) via dataflows.Drivers are up to date. PowerBI is using Microsoft's Spark ODBC driver Version 2.7.6.1014, confirmed via log4j.HybridCloudStoreResultHandler...
- 15 kudos
- 717 Views
- 2 replies
- 0 kudos
API error when creating serverless job "MALFORMED_REQUEST"
I tried executing the job run summit API (/api/2.1/jobs/runs/submit)I have two executions of this API, the first one is working, but the second one don't,I'm getting the following error:{"error_code":"MALFORMED_REQUEST","message":"Invalid JSON given ...
- 717 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @p_hcarvalho, Thanks for your question, here's a corrected version of your JSON payload, can you try it? {"run_name": "my-other-run-name","tasks": [{"task_key": "my-other-task-name","notebook_task": {"notebook_path": "my/notebook/path","source": "...
- 0 kudos
- 789 Views
- 2 replies
- 0 kudos
Speeding up Command Execution API
Hello,I'm using the Databricks Command Execution api to run some spark computations on a dedicated cluster and return the results. I would ideally want the results quickly, especially since the spark computations needed take less that 0.1s.However wh...
- 789 Views
- 2 replies
- 0 kudos
- 0 kudos
My first thought concurs with @Alberto_Umana, the only time I've seen queuing like that is when the cluster is not running. Make sure you have the correct warehouse_id configured in your API calls.
- 0 kudos
- 804 Views
- 1 replies
- 0 kudos
'dbutils.jobs.taskValues.get' taking debug value in workflow, instead of actual value being set
Hi,I am trying to pass and set values from one wheel into another wheel in databricks workflow.I have used 'dbutils.jobs.taskValues.get' and 'dbutils.jobs.taskValues.set'I have used 'dbutils.jobs.taskValues.get' in second task and made sure to keep d...
- 804 Views
- 1 replies
- 0 kudos
- 0 kudos
It seems like the issue you're encountering is related to the debugValue parameter being used instead of the actual value when calling dbutils.jobs.taskValues.get. This behavior is expected when the notebook is run outside of a job context, as the de...
- 0 kudos
- 775 Views
- 1 replies
- 0 kudos
How to get the databricks support contract
We are trying to get the databricks support contract but have no luck. After moving here and there we found this email address gtmops@databricks.com to get support contract but its been 3 weeks and multiple emails from our side but no reply. What's t...
- 775 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Sadam97, Please use help@databricks.com: https://docs.databricks.com/en/resources/support.html
- 0 kudos
- 7658 Views
- 3 replies
- 8 kudos
Drive memory utilization cleanup
Issue Summary:When running multiple jobs on the same compute cluster, over time, I see an increase in memory utilization that is seemingly never fully released, even when jobs finish. This eventually leads to some jobs stalling out as memory hits the...
- 7658 Views
- 3 replies
- 8 kudos
- 8 kudos
I'm encountering something similar. Immediately upon starting a cluster and triggering a job run, my memory usage jumps from 0 to about 20GB used and 15GB cached (see the attached screenshot). The data I am working with should be very small (less tha...
- 8 kudos
- 1508 Views
- 2 replies
- 3 kudos
Working with Pandas through Abfss
Hi, I am unable to read and write pandas datframe through abfss protocol , Is there a work around for this , I do not want to store my files in dbfs
- 1508 Views
- 2 replies
- 3 kudos
- 3 kudos
you can use volumes and mount the abfss location to unity catalog and can access the files present in azure in databricks. Regards,Avinash N
- 3 kudos
- 1538 Views
- 3 replies
- 3 kudos
Resolved! Lakehouse Federation
I use Lakehouse Federation to connect hive metastore (local VM) with metastore database type mysqlIt's can see database and table in hive but when I query data session continue running without failed or success.Do I have to migrate data to ADLs which...
- 1538 Views
- 3 replies
- 3 kudos
- 3 kudos
As for Lakehouse Federation, only some sources are supported as of now. In that, connecting to HiveMetastore data which is in local VM is not supported, so you can migrate that data to ADLS and mount that as external location in unity catalog and que...
- 3 kudos
- 1534 Views
- 2 replies
- 3 kudos
Resolved! How to Apply Encryption Function to a Specific Column
Hello!I would like to apply a function to encrypt a specific column. The UDF is as follows: DROP FUNCTION IF EXISTS EncryptColumn; CREATE FUNCTION EncryptColumn (key_name STRING, encryptcolumn STRING) RETURN base64(aes_encrypt(encryptcolumn, key_nam...
- 1534 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @MadhuB,The method you provided is feasible, and I later finded other ways to apply UDF:UPDATE table_name SET column_name = EncryptColumn(key_name, column_name) Thank you!
- 3 kudos
- 995 Views
- 2 replies
- 0 kudos
Using spark.read.json with a {} literal in my path
I am pulling data from an S3 bucket using spark.read.json like thiss3_uri = "s3://snowflake-genesys/v2.outbound.campaigns.{id}/2025-01-22/00/" df = spark.read.json(s3_uri)My s3 url has the {id} in the file path. I have used r"s3://snowflake-ge...
- 995 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @johngabbradley, Would below approach work for you? s3_uri = "s3://snowflake-genesys/v2.outbound.campaigns.{id}/2025-01-22/00/"files = dbutils.fs.ls(s3_uri)file_paths = [file.path for file in files]df = spark.read.json(file_paths)
- 0 kudos
- 1259 Views
- 1 replies
- 0 kudos
- 1259 Views
- 1 replies
- 0 kudos
- 0 kudos
If you try in the incognito mode does it shows same error?
- 0 kudos
- 1519 Views
- 1 replies
- 0 kudos
How to refer to repository directories in Workflows
HiI need to refer to a configuration file which resides in a separate directory than the script. The paths and execution withing a notebook/python files work fine. When script is scheduled and uses code repository the directory names look obfuscated ...
- 1519 Views
- 1 replies
- 0 kudos
- 0 kudos
can you try with relative path?like you get your current path within test_script.py with command like:cur_path = os.getcwd() then get the path to config.yaml with a relative path like:config_path = os.path.abspath(os.path.join(cur_path, f"../config/c...
- 0 kudos
- 2045 Views
- 6 replies
- 0 kudos
Performance issue when reading data from a view.
Hi All,We are facing some performance issue and I need your help to know what could be the best approach to follow here.Existing: For each region, we have view(Reg1_View,Reg2_View..) to pull data from table( we don't have direct access to table).And ...
- 2045 Views
- 6 replies
- 0 kudos
- 0 kudos
Does any table hold data of all region 1. if yes. Get a Materialized view created (replacing all_reg_view)2. i see you already tried creating a staging table replacing the all_reg_view. Try creating cluster key along with partition.Cluster key on the...
- 0 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
