- 3061 Views
- 3 replies
- 0 kudos
DAB run
Hello All,I am running this command : databricks bundle run -t dev dltPpl_job --debugBundle name: dltPpl. Bundle was generated using: databricks bundle init --target devError message: Error: exit status 1Failed to marshal state to json: unsupported a...
- 3061 Views
- 3 replies
- 0 kudos
- 0 kudos
@Retired_mod I was trying 'pip install --upgrade databricks' before but after upgrading the version with 'brew upgrade databricks it worked. Thanks for the help!
- 0 kudos
- 911 Views
- 2 replies
- 0 kudos
spark config not working in job cluster
I am trying to rename a delta table like this:spark.conf.set("spark.databricks.delta.alterTable.rename.enabledOnAWS", "true")spark.sql("ALTER TABLE db1.rz_test5 RENAME TO db1.rz_test6") The data is on aws s3, that's why I have to use spark config in ...
- 911 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello,it would be interesting to test with the same runtime version.Does it work with a job running on a 13.3 ?
- 0 kudos
- 46663 Views
- 15 replies
- 8 kudos
Error Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient - while trying to create database
Hello All,I get the org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient while trying to create a database scr...
- 46663 Views
- 15 replies
- 8 kudos
- 908 Views
- 2 replies
- 3 kudos
Resolved! Why does my job run on shared compute instead of job compute?
I have configured a job using `databricks.yml````resources: jobs: my_job: name: my_job tasks: - task_key: create_feature_tables job_cluster_key: my_job_cluster spark_python_task: python_file: ../src/c...
- 908 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @cm04 ,You can try to upgrade CLI to newest version. I've seen similiar issue before and upgrading CLI was a solution back then.Solved: Yml file replacing job cluster with all-purpose cl... - Databricks Community - 72248
- 3 kudos
- 5808 Views
- 4 replies
- 3 kudos
Resolved! Inline comment next to un-tickmarked SET statement = Syntax error
Running this code in databricks SQL works great:SET USE_CACHED_RESULT = FALSE; -- Result: -- key value -- USE_CACHED_RESULT FALSEIf I add an inline comment, however, I get a syntax error:SET USE_CACHED_RESUL...
- 5808 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi, I'm getting the same error when trying to execute statement through API "statement": "SET `USE_CACHED_RESULT` = FALSE; SELECT COUNT(*) FROM TABLE" Every combination fail "status": { "state": "FAILED", "error": { "e...
- 3 kudos
- 2307 Views
- 5 replies
- 1 kudos
How to avoid iteration/loop in databricks in the given scenario
Hi all, I need your input.I am new to Databricks and working with a dataset that consists of around 10,000 systems, each containing approximately 100 to 150 parts. These parts have attributes such as name, version, and serial number. The dataset size...
- 2307 Views
- 5 replies
- 1 kudos
- 16807 Views
- 12 replies
- 10 kudos
Connecting SQL Server (on-premise) to Databricks via jdbc:sqlserver
Is it possible to connect to SQL Server on-premise (Not Azure) from Databricks?I tried to ping my virtualbox VM (with Windows Server 2022) from within Databricks and the request timed out.%sh ping 122.138.0.14This is what my connection might look l...
- 16807 Views
- 12 replies
- 10 kudos
- 10 kudos
I tried to connect to localhost sql server through databricks community edition, but it failed. I have created an IP rule on port 1433 allowed inbound connection from all public network, but still didn't connect. I tried locally using python its work...
- 10 kudos
- 1638 Views
- 4 replies
- 2 kudos
Resolved! How to create a DLT pipeline with SQL statement
I need a DLT pipeline to create a materialized view for fetching event logs. All the ways below I tried are failed:Attach a notebook with pure SQL inside: No magic cell like `%sql` are failedAttach a notebook with `spark.sql` python code: Failed beca...
- 1638 Views
- 4 replies
- 2 kudos
- 2 kudos
After just finishing my last reply, I realized what’s wrong with my code: I should use “file” property instead of “notebook” in the libraries section.It works now. Thank you guys, you are my rubber duck!
- 2 kudos
- 1253 Views
- 1 replies
- 0 kudos
Getting PermissionDenied in SDK When Updating External Location Isolation Mode
Using the Databricks SDK for Python in a notebook in a Databricks workspace, I'm creating an external location and then attempting to update the isolation mode and workspace bindings associated with the external location. The step to create the exter...
- 1253 Views
- 1 replies
- 0 kudos
- 0 kudos
Let me clean-up these cells for better readability:%pip install databricks-sdk --upgradedbutils.library.restartPython()from databricks.sdk import WorkspaceClient from databricks.sdk.service import catalogw = WorkspaceClient()# This works without issu...
- 0 kudos
- 4324 Views
- 2 replies
- 0 kudos
Obtain the source table version number from checkpoint file when using Structured Streaming
Hello!I'm using Structured Streaming to write to a delta table. The source is another delta table written with Structured Streaming as well. In order to datacheck the results I'm attempting to obtain from the checkpoint files of the target table the ...
- 4324 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @Retired_mod, thank you for your answer.I'm a bit confused here because you seem to be describing the opposite behavior of what I've seen in our checkpoint files.Here I repost my examples to try to understand better.First checkpoint file:{"sour...
- 0 kudos
- 4978 Views
- 5 replies
- 9 kudos
Databricks Cluster Web terminal different permissions with tmux and xterm.
I am launching web terminal on my databricks cluster and when I am using the ephemeral xterm instance I am easily able to navigate to desired directory in `Workspace` and run anything... for example `ls ./` When I switch to tmux so that I can preserv...
- 4978 Views
- 5 replies
- 9 kudos
- 9 kudos
Hey there, fellow data explorer pals! I totally get your excitement when launching that web terminal on your Databricks cluster and feeling the power of running commands like 'ls ./' in the ephemeral xterm instance. It's like traversing the vast univ...
- 9 kudos
- 1400 Views
- 2 replies
- 2 kudos
Resolved! alter DLT Materialized View alter column set MASK
I am trying to mask a column on a DLT materialized view - this is created using DLT syntax. I am not able set the column masking after creation. Appreciate any workaround.alter DLT Materialized View alter column set MASK
- 1400 Views
- 2 replies
- 2 kudos
- 27226 Views
- 15 replies
- 12 kudos
Resolved! How to query delta lake using SQL desktop tools like SSMS or DBVisualizer
Is there a way to use sql desktop tools? because delta OSS or databricks does not provide desktop client (similar to azure data studio) to browse and query delta lake objects.I currently use databricks SQL , a webUI in the databricks workspace but se...
- 27226 Views
- 15 replies
- 12 kudos
- 12 kudos
DSR is Delta Standalone Reader. see more here - https://docs.delta.io/latest/delta-standalone.htmlIts a crate (and also now a py library) that allows you to connect to delta tables without using spark (e.g. directly from python and not using pyspa...
- 12 kudos
- 831 Views
- 2 replies
- 0 kudos
Upload of .bin file >400mb
I try to upload to local workspace folder with .bin extension.It is required to have it locally.I tried load from DBFS, but loading files over 265mb is not allowed with cluster. I tried to upload manually but failed with same error "OSError: [Errno5]...
- 831 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello Slash,Thank you for your response. I'm encountering the same issue as described. I tried running the provided code in my Databricks workspace, but I received an error. My question is how the script is expected to access local files, especially ...
- 0 kudos
- 1742 Views
- 4 replies
- 0 kudos
Resolved! Discrepancies in Data Engineering GitHub Repositories and Errors in Databricks Notebooks
Hi team,I am writing to express my concerns regarding the recent Databricks webinar on "Data Intelligence with Databricks". During the session, there was mention of two GitHub repositories associated with the notebooks being used as mentioned below.1...
- 1742 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Meghana_Vasavad could you please file a ticket with Databricks SupportThey will help you with this request.
- 0 kudos
| User | Count |
|---|---|
| 1613 | |
| 771 | |
| 349 | |
| 286 | |
| 253 |
