- 905 Views
- 3 replies
- 5 kudos
Resolved! Unable to Delete Failed Databricks Job VMs in Azure
My Job Compute had trouble on starting the cluster, acusing "Unexpected failure while waiting for the cluster (xxxx) to be ready: Cluster 'xxxx' is unhealthy"After multiple retries, a new error message appeared:"Operation could not be completed as it...
- 905 Views
- 3 replies
- 5 kudos
- 5 kudos
UPDATE: Before opening the Support Ticket, the machines suddently disappeared. I deleted the Jobs definitions with its JobClusters definitions, and maybe that solved it, or after some hours the machines were cleaned. Not sure what cleaned it.Also I n...
- 5 kudos
- 3050 Views
- 3 replies
- 1 kudos
Resolved! Unable to unpack or read rar file
Hi everyone,I'm encountering an issue with the following code when trying to unpack or read a RAR file in Databricks: with rarfile.RarFile(s3_path) as rf: for file_info in rf.infolist(): with rf.open(file_info) as file: file_c...
- 3050 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Walter_C,I am also using this unrar utility but the problem it is a proprietary software and i am working for a client and this license could cause issues. What is the alternative to unrar so that we eliminate the risk of any legal compliance.
- 1 kudos
- 2008 Views
- 5 replies
- 1 kudos
Resolved! How to design Airship Integration with Azure Databricks
Hello,I have to push data from Airship and persists it to Delta tables. I think We can used SFTP , May someone please help me how to design the inbound part , it using SFTP on Airship end to push file on ADLS Gen2.networking and security consideratio...
- 2008 Views
- 5 replies
- 1 kudos
- 1 kudos
Inbound Flow DesignEnable SFTP on the ADLS Gen2 (or Azure Blob Storage) account;Generate and register an SSH public/private key pair with Airship, enter your SFTP endpoint credentials (username, host, port, key) in Airship’s settings to authenticate ...
- 1 kudos
- 2726 Views
- 10 replies
- 5 kudos
Resolved! CONTAINER_LAUNCH_FAILURE
Hello everyone!I need some help, unable to get cluster up and running. I did try creating classic compute but fails, is there any limit to use databricks community edition? Error here: { "reason": { "code": "CONTAINER_LAUNCH_FAILURE", "type...
- 2726 Views
- 10 replies
- 5 kudos
- 5 kudos
To all legacy community edition is working fine if you use dbr <= 15.4 for both general and ML modes.I think legacy community still far more better than free edition. I was selecting >15.4 DBR Thank you.
- 5 kudos
- 664 Views
- 2 replies
- 0 kudos
Serverless performance_target option is not available for one time jobs
Why this option https://docs.databricks.com/api/workspace/jobs/create#performance_target does not available for one-time runs https://docs.databricks.com/api/workspace/jobs/submit ?
- 664 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello @SiarheiSintsou! The performance_target isn’t currently supported in the SubmitRun API. However, it would be helpful if you could submit a feature request here.
- 0 kudos
- 4267 Views
- 2 replies
- 0 kudos
Identifying Full Refresh vs. Incremental Runs in Delta Live Tables
Hello Community,I am working with a Delta Live Tables (DLT) pipeline that primarily operates in incremental mode. However, there are specific scenarios where I need to perform a full refresh of the pipeline. I am looking for an efficient and reliable...
- 4267 Views
- 2 replies
- 0 kudos
- 0 kudos
Hello,There are two ways to determine whether a DLT pipeline is running in Full Refresh or Incremental mode:DLT Event Log SchemaThe details column in the DLT event log schema includes information on "full_refresh". You can use this to identify whethe...
- 0 kudos
- 1346 Views
- 2 replies
- 0 kudos
Resolved! ModuleNotFoundError: No module named 'databricks.sdk.service.database'
Hi , https://learn.microsoft.com/en-gb/azure/databricks/oltp/sync-data/sync-table?source=docs#python-sdk The module cannot be found. The cluster is as screenshot and the code is from docs. Best regards,
- 1346 Views
- 2 replies
- 0 kudos
- 1004 Views
- 2 replies
- 2 kudos
Resolved! Deleting database tables from a catalog
Hello,I am exploring lakebase and sync the table to postgre catalog. When I try to drop a table, I got this errorof kind: Some(CATALOG_MANAGED_POSTGRESQL) is not supported. How to fix it? I have all the privileges. Thanks.Best regards
- 1004 Views
- 2 replies
- 2 kudos
- 2 kudos
Hello @zyang! Could you please check if you’re following the steps outlined in the doc? First, you need to delete the synced table in Unity Catalog and then drop it from PostgreSQL using the DROP TABLE command.
- 2 kudos
- 1076 Views
- 2 replies
- 0 kudos
Facing difficulty in connecting access db from databricks
Unable to connect Microsoft Access DB in Databricks, I tried with JDBC & Pyodbc connectors but I'm not able to read the file from accessdb. Please suggest or help to resolve this issue
- 1076 Views
- 2 replies
- 0 kudos
- 0 kudos
Installed office365-REST-Python-Client==2.5.13UcanaccessDriver # File location and typefile_location = "/FileStore/tables/ABU.accdb"file_type = "accdb"import pyodbc# Connection string (Windows only, or if driver is available)conn_str = ( # Change t...
- 0 kudos
- 5382 Views
- 8 replies
- 3 kudos
Resolved! AutoLoader Pros/Cons When Extracting Data
Hi there, I am interested in using AutoLoader, but I'd like to get a bit of clarity if it makes sense in my case. Based on examples I've seen, an ideal use-case for AutoLoader is when we have some kind of landing path where we expect raw files to arr...
- 5382 Views
- 8 replies
- 3 kudos
- 3 kudos
I am in the position to have more and more use-cases for AutoLoader. This might be an obvious question but if I understand correctly, Christian's use-case without the landing stage could still be an auto-loader scenario if the extracts from source sy...
- 3 kudos
- 553 Views
- 1 replies
- 0 kudos
Fetching data from Databricks server with delta sharing limits to 100 records
Hi,I'm trying to fetch a table from a Databricks instance hosted in Azure using delta sharing python library. The delta sharing library always returns a dataframe of length 100 when fetching table data. I tested all the tables shared with me. The ins...
- 553 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @uddipak ,Maybe internally load_as_pandas has some default limit? Can you try to set limit explicitly?import delta_sharing client = delta_sharing.SharingClient("...") client.list_all_tables() # this works fine . . . delta_sharing.load_as_pandas(t...
- 0 kudos
- 5629 Views
- 2 replies
- 3 kudos
Resolved! Liquid clustering with boolean columns
Hi community Is it possible to use boolean columns as cluster keys for liquid clustering on Delta Tables? I've been trying to set a boolean column as cluster key since it's one of my most common queries when reading from the table. I'm getting the er...
- 5629 Views
- 2 replies
- 3 kudos
- 3 kudos
I saw that boolean columns are not supported by liquid clustering, too. Why? In any case:By now there is an error that can get raised called DELTA_CLUSTERING_COLUMNS_DATATYPE_NOT_SUPPORTED.There is also by now documentation of column datatypes that...
- 3 kudos
- 10651 Views
- 12 replies
- 3 kudos
Delay in files update on filesystem
Hi, I noticed that there is quite a significant delay (2 - 10s) between making a change to some file in Repos via Databricks file edit window and propagation of such change to the filesystem. Our engineers and scientists use YAML config files. If the...
- 10651 Views
- 12 replies
- 3 kudos
- 3 kudos
Similar issue where opening up the git interface can take 30+ seconds to show changes.
- 3 kudos
- 2610 Views
- 3 replies
- 4 kudos
Resolved! TABLE_OR_VIEW_ALREADY_EXISTS when using saveAsTable with append mode
I ran into an unexpected error while writing a DataFrame to a managed table df.write \ .mode("append") \ .option("overwriteSchema", "true") \ .option("mergeSchema", "true") \ .saveAsTable(bsr_sink_table_fqn)Instead of appending, Spark attemp...
- 2610 Views
- 3 replies
- 4 kudos
- 4 kudos
Thanks @szymon_dybczak and @WiliamRosa for the reply. I will probably keep mergeSchema then. If the options are incompatible or might cause issues I wish there was a warning of sorts.
- 4 kudos
- 3655 Views
- 5 replies
- 3 kudos
Resolved! Incrementalization issue in Materialized views
I am trying to implement the incremental updates to the Materialized views. The source is the Could you tell how to resolve the following issue? {  "planning_information": {    "technique_information": [      {        "maintenance_type": "MAINTENANCE...
- 3655 Views
- 5 replies
- 3 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 565 | |
| 349 | |
| 287 |
