Databricks

The configs are for cluster:Worker Type & Driver type : Standard_D16ads_v5RUs for Cosmos : 1.5L

Thanks for your comments. The problem turned out to be the compute resource not having unity catalog enabled.

Hi @Sagas, Let’s address your questions regarding Azure Databricks, SparkR, and Sparklyr. History in Azure Databricks: Each operation that modifies a Delta Lake table creates a new table version. You can use history information to audit operation...

Can you run the following command in a notebook using the same cluster you are using to connect:%sh nc -vz <hostname> <port> This test will confirm us if we are able to communicate with the SQL server by using the port you are defining to connect. If...

Hello Joaquim,Your issue might be related to the access mode of your cluster which probably has been selected to be Shared Access Mode.Shared cluster only allows Python, SQL and Scala languages, you might need to change the access mode to be Single U...

Hi @enkefalos-commu , I apologize for the inconvenience you’re facing. Let’s troubleshoot this issue together. Here are some steps you can take: Check Your Python Environment: Ensure that you are using a compatible Python environmentTransformer...

Hi @subha2, To dynamically validate the data quality (DQ) rules for tables configured in a metadata-driven system using PySpark, you can follow these steps: Define Metadata for Tables: First, create a metadata configuration that describes the rules ...

Hi @Phani1, When configuring a job cluster for 24/7 operation, it’s essential to consider cost, performance, and scalability. Here are some recommendations based on your specified parameters: Data Volumes: Analyze your data volumes carefully. If...

Hi @agarg, As of now, the Databricks REST API does not directly provide a specific endpoint to fetch workspace mount points or mount information. However, you can achieve this by executing SQL queries on Databricks SQL Warehouse using the Databricks ...

Hi @Sikki Good day! There was an issue and it was fixed recently. Could you please confirm if you are still facing the issue? Best regards,

I got it eventually working with a combination of:from databricks.sdk.runtime import *spark.sparkContext.addPyFile("/path/to/your/file")sys.path.append("path/to/your")   

Hi @pfpmeijers, As of now, Databricks primarily operates as a unified, open analytics platform for constructing, deploying, sharing, and maintaining enterprise-grade data, analytics, and AI solutions at scale. It seamlessly integrates with cloud stor...

Hi,were you able to get any useful help on this?

Does cloning take considerably less time then recreating the tables?Can I resume append operations to a cloned table?