- 1348 Views
- 1 replies
- 0 kudos
Best Approach to Store Data in Azure Gov Cloud Workspace without Unity Catalog
Our team is using a workspace Azure Gov Cloud. We would like to download files from an external source into our Workspace. Since Unity Catalog is not enabled in Azure Gov Cloud, we’re looking for the best approach for securely storing data in our wor...
- 1348 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello!For securely storing sensitive data in your Azure Gov Cloud workspace without Unity Catalog, use Azure Blob Storage with encryption, Azure Data Lake Storage with AAD access control, or Azure Key Vault for secrets. Managed identities can also he...
- 0 kudos
- 9084 Views
- 6 replies
- 5 kudos
SAP Business Object(BO) Integration with Databricks
Hi Team,We are doing an analysis on SAP Business object to connect with databricks and built a report on top of the data in the data lakehouse. In our current architecture we have delta tables on top of S3 storage. Please let us know any connectors/d...
- 9084 Views
- 6 replies
- 5 kudos
- 5 kudos
Hi @HB83 , Can I know which version of BO are you using? We have a similar requirement.
- 5 kudos
- 9912 Views
- 4 replies
- 2 kudos
Resolved! How to use autoloader with csv containing spaces in attribute names?
I am attempting to use autoloader to add a number of csv files to a delta table. The underlying csv files have spaces in the attribute names though (i.e. 'Account Number' instead of 'AccountNumber'). When I run my autoload, I get the following error ...
- 9912 Views
- 4 replies
- 2 kudos
- 2 kudos
@Hubert Dudek​ thanks for your response! I was able to use what you proposed above to generate the schema. The issue is that the schema sets all attributes to STRING values and renames them numerically ('_c0', '_c1', etc.). Although this allows us to...
- 2 kudos
- 20337 Views
- 12 replies
- 7 kudos
dataframe takes unusually long time to save as a delta table using sql for a very small dataset with 30k rows. It takes around 2hrs. Is there a solution for this problem?
I am trying to save a dataframe after a series of data manipulations using Udf functions to a delta table. I tried using this code( df .write .format('delta') .mode('overwrite') .option('overwriteSchema', 'true') .saveAsTable('output_table'))but this...
- 20337 Views
- 12 replies
- 7 kudos
- 7 kudos
You should also look into the sql plan if the writing phase is indeed the part that is taking time. Since spark works on lazy evaluation, there might be some other phase that might be taking time
- 7 kudos
- 3890 Views
- 1 replies
- 0 kudos
SQL function refactoring into Databricks environment
Hello all,I'm currently working on importing some SQL functions from Informix Database into Databricks using Asset Bundle deploying Delta Live Table to Unity Catalog. I'm struggling importing a recursive one, there is the code :CREATE FUNCTION "info...
- 3890 Views
- 1 replies
- 0 kudos
- 14454 Views
- 7 replies
- 4 kudos
kafkashaded.org.apache.kafka.common.errors.TimeoutException: topic-downstream-data-nonprod not present in metadata after 60000 ms.
I am facing an error when trying to write data on Kafka using spark stream.#Extract source_stream_df= (spark.readStream .format("cosmos.oltp.changeFeed") .option("spark.cosmos.container", PARM_CONTAINER_NAME) .option("spark.cosmos.read.inferSchema.en...
- 14454 Views
- 7 replies
- 4 kudos
- 4 kudos
What event hub namespace you were using?I had same problem and resolved by changing pricing plan from basic to standard as Kafka apps is not supporting in basic planLet me know if you had anything else. Thanks
- 4 kudos
- 7304 Views
- 6 replies
- 1 kudos
Unity Catalog is set up for multi region
Hello Team, I need some clarification on the below diagram . According to the documentation, the Unity Catalog is set up for each region. If we are using multiple clouds, the diagram shows only one Unity Catalog across regions. Shouldn't there be two...
- 7304 Views
- 6 replies
- 1 kudos
- 1 kudos
Yes, the reason why they are grouped into a single rectangle is probably to show that they both are Unity Catalog enabled. It can indeed be confusing, represented like that.If you want to let them connect to each other, delta sharing or metastore fed...
- 1 kudos
- 12678 Views
- 6 replies
- 0 kudos
Foreign catalog - Connections using insecure transport are prohibited --require_secure_transport=ON
I have added a connection to a MySql database in Azure, and I have created a foreign catalog in Databricks. But when I go to query the database I get the following error;Connections using insecure transport are prohibited while --require_secure_trans...
- 12678 Views
- 6 replies
- 0 kudos
- 0 kudos
I was unable to use serverless compute with Azure MySQL. I did have a meeting with tech support at databricks and the conclusion at the time was - if your Azure MySql is not publically accessible, you cannot connect to it from serverless compute.I'm ...
- 0 kudos
- 2793 Views
- 3 replies
- 1 kudos
Resolved! Error when query event log: Cannot read more than one event logs in the same query
I try to follow the instructions of Monitor Delta Live Tables pipelines to query dlt.expection log.Here is the simple code version I copied from the Querying the event log section:CREATE TEMPORARY LIVE VIEW event_log_raw AS (SELECT * FROM event_log(T...
- 2793 Views
- 3 replies
- 1 kudos
- 1 kudos
H @szymon_dybczak , thank you for the advice. After investigating the problem has been solved. The error message mentioned in the title is not the key, the below error message “A pipeline with a different id is already registered in this Credential S...
- 1 kudos
- 4030 Views
- 3 replies
- 0 kudos
DAB run
Hello All,I am running this command : databricks bundle run -t dev dltPpl_job --debugBundle name: dltPpl. Bundle was generated using: databricks bundle init --target devError message: Error: exit status 1Failed to marshal state to json: unsupported a...
- 4030 Views
- 3 replies
- 0 kudos
- 0 kudos
@Retired_mod I was trying 'pip install --upgrade databricks' before but after upgrading the version with 'brew upgrade databricks it worked. Thanks for the help!
- 0 kudos
- 1589 Views
- 2 replies
- 0 kudos
spark config not working in job cluster
I am trying to rename a delta table like this:spark.conf.set("spark.databricks.delta.alterTable.rename.enabledOnAWS", "true")spark.sql("ALTER TABLE db1.rz_test5 RENAME TO db1.rz_test6") The data is on aws s3, that's why I have to use spark config in ...
- 1589 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello,it would be interesting to test with the same runtime version.Does it work with a job running on a 13.3 ?
- 0 kudos
- 61910 Views
- 15 replies
- 8 kudos
Error Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient - while trying to create database
Hello All,I get the org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient while trying to create a database scr...
- 61910 Views
- 15 replies
- 8 kudos
- 1612 Views
- 2 replies
- 3 kudos
Resolved! Why does my job run on shared compute instead of job compute?
I have configured a job using `databricks.yml````resources: jobs: my_job: name: my_job tasks: - task_key: create_feature_tables job_cluster_key: my_job_cluster spark_python_task: python_file: ../src/c...
- 1612 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @cm04 ,You can try to upgrade CLI to newest version. I've seen similiar issue before and upgrading CLI was a solution back then.Solved: Yml file replacing job cluster with all-purpose cl... - Databricks Community - 72248
- 3 kudos
- 8223 Views
- 4 replies
- 3 kudos
Resolved! Inline comment next to un-tickmarked SET statement = Syntax error
Running this code in databricks SQL works great:SET USE_CACHED_RESULT = FALSE; -- Result: -- key value -- USE_CACHED_RESULT FALSEIf I add an inline comment, however, I get a syntax error:SET USE_CACHED_RESUL...
- 8223 Views
- 4 replies
- 3 kudos
- 3 kudos
Hi, I'm getting the same error when trying to execute statement through API "statement": "SET `USE_CACHED_RESULT` = FALSE; SELECT COUNT(*) FROM TABLE" Every combination fail "status": { "state": "FAILED", "error": { "e...
- 3 kudos
- 3907 Views
- 5 replies
- 1 kudos
How to avoid iteration/loop in databricks in the given scenario
Hi all, I need your input.I am new to Databricks and working with a dataset that consists of around 10,000 systems, each containing approximately 100 to 150 parts. These parts have attributes such as name, version, and serial number. The dataset size...
- 3907 Views
- 5 replies
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
