- 4435 Views
- 1 replies
- 0 kudos
Delta Live Tables and GIT
Notebooks that runs in a delta live table are GIT enabled, but what about the Delta Live Table pipeline?I'm looking for a good way to deploy pipelines from DEV to TEST and TEST to PROD that not just deploy the notebooks but also the pipeline.What pos...
- 4435 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @Henrik , Databricks Asset Bundles would help you do this - https://docs.databricks.com/en/dev-tools/bundles/pipelines-tutorial.html Also this is a wonderful post addressing your query - https://www.databricks.com/blog/applying-software-develop...
- 0 kudos
- 1473 Views
- 2 replies
- 0 kudos
Enable delta sharing oss using pyspark.
I have started the delta sharing server.Using guide on https://github.com/delta-io/delta-sharing.But I am not able to create a profile.How can I correctly create profile and share the table through delta sharing with the data stored in MinIO?
- 1473 Views
- 2 replies
- 0 kudos
- 0 kudos
@gupta_tanmay wrote:I have started the delta sharing server.Using guide on https://github.com/delta-io/delta-sharing.But I am not able to create a profile.How can I correctly create profile and share the table through delta sharing with the data stor...
- 0 kudos
- 1226 Views
- 1 replies
- 0 kudos
Migration to new Tools
Has anyone else taken a migration journey from SAS to a different language/platform?? Love to hear about your journey…
- 1226 Views
- 1 replies
- 0 kudos
- 0 kudos
@cannonball111 Yes, we have done using Travinto Technologies tool
- 0 kudos
- 2422 Views
- 1 replies
- 0 kudos
Use DataBricks migration tool to export query
Dear all,I tried to use Databricks migration tool (https://github.com/databrickslabs/migrate) to migrate objects from one Databricks instance to another. I realized that notebooks, clusters, jobs can be done but queries can not be migrated by this to...
- 2422 Views
- 1 replies
- 0 kudos
- 0 kudos
@yjiao If you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports across platf...
- 0 kudos
- 4523 Views
- 1 replies
- 0 kudos
URGENT - Databricks workspace deletion & recovery
Hi Team,I accidentally deleted our databricks workspace, which had all our artefacts and control plane, and was the primary resource for our team's working environment.Could anyone please help on priority, regarding the recovery/ restoration mechanis...
- 4523 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello data-warriors Note: Users cannot recover the deleted Databricks instance directly from Azure Portal. Deleted databricks instance can be only recovered by opening a support ticket, where our core engineering team will help you to recover the Da...
- 0 kudos
- 3139 Views
- 1 replies
- 0 kudos
Unity catalog Migration
Hi Team,Could you please help me to understand, 1)Why we need to migrate Unity catalog? if we are not migrating what benefits we will not get?2) How to migrate Unity catalog (What all are objects needs to migrate and any tool) ? Regards,Phanindra
- 3139 Views
- 1 replies
- 0 kudos
- 0 kudos
@Phani1 if you're planning to migrate from your current technology to Databricks, Travinto Technologies' Code Converter Tool is here to make the process seamless. This powerful tool enables you to migrate data, ETL workflows, and reports across platf...
- 0 kudos
- 2099 Views
- 2 replies
- 0 kudos
Disable mlflow autologging in a helper notebook
We have a helper function that uses a sklearn estimator. We don't want to to be logged to mlflow.I can dodef myfunc(): import mlflow with mlflow.autolog.ignore: # train model # use model return predictionsBut I get info prints:...
- 2099 Views
- 2 replies
- 0 kudos
- 0 kudos
mlflow.autolog(disable=True, silent=True) fixes the printing. But my other problem with setting autologging back to previous state is still unsolved. I can't find any information about that problem in the docs.
- 0 kudos
- 1866 Views
- 1 replies
- 1 kudos
end-to-end lineage using Unity Catalog
Hi Team,what is the Possibilities of getting end-to-end lineage using Unity CatalogRegards,Phanindra
- 1866 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello Phanindra, Inside Databricks ecosystem, you can easily get end-to-end lineage. The data lineage will be build automatically as you ingest and transform data inside the lakehouse. Here is a technical view on how this is achieved: On the other ha...
- 1 kudos
- 2469 Views
- 1 replies
- 0 kudos
Data Masking issue
Hi Team,After successfully creating a group and populating it with members, I am facing the problem of the is_member() function always returning False, despite the user being part of the group. This issue is specific to my sandbox Databricks Account....
- 2469 Views
- 1 replies
- 0 kudos
- 0 kudos
Can you post your code? I use the 'is_account_group_member' in my code as follows. Also note there is a alternative to Dynamic Views that is preferred. It's named 'Row filtering and Column masking'. Go here to learn more: https://docs.databricks...
- 0 kudos
- 3898 Views
- 6 replies
- 0 kudos
databricks asset bundle error-terraform.exe": file does not exist
Hi,I am getting below error while I am deploying databricks bundle using azure devops release 2024-07-07T03:55:51.1199594Z Error: terraform init: exec: "xxxx\\.databricks\\bundle\\dev\\terraform\\xxxx\\.databricks\\bundle\\dev\\bin\\terraform.exe": ...
- 3898 Views
- 6 replies
- 0 kudos
- 0 kudos
Using Git Bash for bash execution and to setup variables we did this by going to control panel -> system -> environment varibales
- 0 kudos
- 3241 Views
- 3 replies
- 0 kudos
Foreign table to delta streaming table
I want to copy a table from a foreign catalog as my streaming table. This is the code I used but I am getting error: Table table_name does not support either micro-batch or continuous scan.; spark.readStream .table(table_name) ...
- 3241 Views
- 3 replies
- 0 kudos
- 0 kudos
What is the underlying type of the table you are trying to stream from? Structured Streaming does not currently support streaming reads via JDBC, so reading from MySQL, Postgres, etc are not supported. If you are trying to perform stream ingestion fr...
- 0 kudos
- 2356 Views
- 1 replies
- 0 kudos
cross join issue generating surrogate keys in delta table
I used below code to populate target table, it is working as expected but expect for surrogatekey column. After I inserted dummy entry -1 and did run merge code, it is generating the numbers in surrogatekey column like this 1,3,5,7,9 odd numbers, it ...
- 2356 Views
- 1 replies
- 0 kudos
- 0 kudos
This is expected. Identity columns can have gaps, or per the documentation: Values assigned by identity columns are unique and increment in the direction of the specified step, and in multiples of the specified step size, but are not guaranteed to b...
- 0 kudos
- 1931 Views
- 4 replies
- 2 kudos
Expose Databricks Cluster Logs to Grafana
I am trying to send Databricks cluster logs to Grafana using an init_script where I define the system journals to consume. The issue I am facing is that I cannot get the driver logs, standard error, and output to reach Grafana. Is there something spe...
- 1931 Views
- 4 replies
- 2 kudos
- 1349 Views
- 3 replies
- 0 kudos
Certification exam got suspended
Hello Team,My Data engineer associate exam got suspended within seconds of starting without any reason. After starting the exam screen got paused just after 10-20 seconds there was a notice that someone will contact you but no one contacted till the ...
- 1349 Views
- 3 replies
- 0 kudos
- 0 kudos
You need to raise a ticket with Databricks to get to a resolution: https://help.databricks.com/s/contact-us?ReqType=training%22%20%5Ct%20%22_blank
- 0 kudos
- 1363 Views
- 1 replies
- 0 kudos
Liquid Clustering
How can I use column for liquid clustering that is not in first 32 column of my delta table schema.
- 1363 Views
- 1 replies
- 0 kudos
- 0 kudos
We can only specify columns with statistics collected for clustering keys. By default, the first 32 columns in a Delta table have statistics collected. See Specify Delta statistics columns. We can use the below workaround for your use case: 1. Use th...
- 0 kudos
