- 8090 Views
- 3 replies
- 1 kudos
How to export/clone Databricks Notebook without results via web UI?
When a Databricks Notebook exceeds size limit, it suggests to `clone/export without results`. This is exactly what I want to do, but the current web UI does not provide the ability to bypass/skip the results in either the `clone` or `export` context...
- 8090 Views
- 3 replies
- 1 kudos
- 1 kudos
Thank you @Yeshwanth for the response. I am looking for a way without clearing up the current outputs. This is necessary because I want to preserve the existing outputs and fork off another notebook instance to run with few parameter changes and come...
- 1 kudos
- 25068 Views
- 10 replies
- 4 kudos
Resolved! Databricks Asset Bundle (DAB) from a Git repo?
My earlier question was about creating a Databricks Asset Bundle (DAB) from an existing workspace. I was able to get that working but after further consideration and some experimenting, I need to alter my question. My question is now "how do I create...
- 25068 Views
- 10 replies
- 4 kudos
- 17981 Views
- 7 replies
- 4 kudos
Resolved! Databricks SQL endpoint as Linked Service in Azure Data Factory
We have a special endpoint that grants access to delta tables and we want to know if we can use SQL endpoints as a linked service in ADF.If yes then which ADF-linked service would be suitable for this?Appreciate your support on this.
- 17981 Views
- 7 replies
- 4 kudos
- 4 kudos
Azure Databricks Delta Lake (Dataset) uses a linked service that can only connect to a All Purpose/Interactive cluster.If you want to use the SQL Endpoint, you would need a Self Hosted Integration Runtime for ADF with Databricks ODBC driver Installed...
- 4 kudos
- 9639 Views
- 4 replies
- 1 kudos
DLT SCD type 2 in bronze, silver and gold? Is it possible?
I have a question related to when we are using #DLT. Let me try to describe the DLT problem: Objective: Process data end-to-end (bronze, silver gold) using DLTWant bronze to hold a complete raw replica of the data, leveraging apply_changes SCD to wr...
- 9639 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi,At the moment, my process is:-I am using ETL (Data Factory) to land paquet files in a raw landing zone. I keep all the source data here. So I can fully rebuild the data if I need to.i.e.source_system/schema/table/loadon_year=2024/loadon_month=08/l...
- 1 kudos
- 6358 Views
- 1 replies
- 0 kudos
TypeError: ColSpec.__init__() got an unexpected keyword argument 'required'
Hi Team, one of my customer is facing the below issue.. Anyone faced this issue before ? Any help would be appreciated.import mlflowmlflow.set_registry_uri("databricks-uc")catalog_name = "system"embed = mlflow.pyfunc.spark_udf(spark, f"models:/system...
- 6358 Views
- 1 replies
- 0 kudos
- 0 kudos
Upgrade the version of the mlflow package. In 2.7.1 there is no such parameter. https://mlflow.org/docs/2.7.1/python_api/mlflow.types.htmlWhile in the latest (2.17.0) there is such https://mlflow.org/docs/2.17.0/python_api/mlflow.types.html%pip insta...
- 0 kudos
- 2576 Views
- 2 replies
- 1 kudos
Resolved! autoloader error using unity catalog
Hello! I'm new on Databricks and I'm exploring some of its features.I've successfully configured a workspace with unity catalog, one external storage location (ADLSg2) and the associated storage credential. I provided all privileges for all account u...
- 2576 Views
- 2 replies
- 1 kudos
- 1 kudos
hey @garf could you please try to create an external volume using your external location and then use the file path in the volume as the input file path?
- 1 kudos
- 2032 Views
- 2 replies
- 1 kudos
Access issue in iPad Air m2
I am trying to run databricks community edition in iPad Air m2. It’s showing me blank page while logging, but in my android phone it’s working fine. Is there any compatibility issue with ios ? Please help me
- 2032 Views
- 2 replies
- 1 kudos
- 1 kudos
Thank you for your response. While I understand that mobile browsers are currently not officially supported by Databricks, I believe this approach does not align with the expectations and needs of modern users. In today's digital landscape, it is ess...
- 1 kudos
- 2187 Views
- 3 replies
- 0 kudos
Call Azure Cognitive Services API from Notebook using Azure Entra ID that is logged on to Databricks
I would like to call Azure Cognitive Services API from Notebook using Azure Entra ID that is logged on to Databricks. The cognitive services key is not available as local authentication for cognitive services is not enables.
- 2187 Views
- 3 replies
- 0 kudos
- 0 kudos
and what if you do not use a token but pass the credential in the cognitive services client (like Text Analytics Client)?It could also be a networking/firewall setting that prevents you from calling ACS.Permissions can also be the cause.
- 0 kudos
- 14410 Views
- 7 replies
- 4 kudos
File Arrival Trigger
We are using Databricks in combination with Azure platforms, specifically working with Azure Blob Storage (Gen2). We frequently mount Azure containers in the Databricks file system and leverage external locations and volumes for Azure containers.Our ...
- 14410 Views
- 7 replies
- 4 kudos
- 4 kudos
@KrzysztofPrzyso Yes, we are now following this approach as an alternative solution, which involves combining Autoloader with Databricks File Arrival. In this case, you don't need to run cluster all the time instead use below Autoloader config. Pass ...
- 4 kudos
- 11226 Views
- 6 replies
- 0 kudos
applyInPandas started to hang on the runtime 13.3 LTS ML and above
Hello, recently I've tried to upgrade my runtime env to the 13.3 LTS ML and found that it breaks my workload during applyInPandas.My job started to hang during the applyInPandas execution. Thread dump shows that it hangs on direct memory allocation: ...
- 11226 Views
- 6 replies
- 0 kudos
- 7021 Views
- 6 replies
- 1 kudos
Where are materialized view generated by Delta Live Table stored?
I am trying to compare tables created by DBT in Catalog vs the materialized view generated by Delta Live Table, and I noticed that the dbt generated table has Storage Location information and It points to a physical storage location however the mater...
- 7021 Views
- 6 replies
- 1 kudos
- 1 kudos
It would be good if Databricks would confirm the behaviour, rather than requiring us to make assumptions like this.
- 1 kudos
- 3025 Views
- 1 replies
- 1 kudos
Resolved! File Arrival Trigger in Azure Databricks
We are using Databricks in combination with Azure platforms, specifically working with Azure Blob Storage (Gen2). We frequently mount Azure containers in the Databricks file system and leverage external locations and volumes for Azure containers.Our ...
- 3025 Views
- 1 replies
- 1 kudos
- 1 kudos
@ShresthaBaburam We inquired about this a few days ago and checked with Databricks. They were working on the issue, but no ETA was provided. You can find more details here: Databricks Community Link.However, to address this use case, we followed the ...
- 1 kudos
- 2105 Views
- 1 replies
- 0 kudos
Sync table A to table B, triggered by any change in table A.
Hey,I'm trying to find a way to sync table A to table B whenever table A is written to. just with a trigger on write.I want to avoid using any continuous runs or schedules.Trying to get this to work inside Databricks, without having to use any outsid...
- 2105 Views
- 1 replies
- 0 kudos
- 0 kudos
@Yarden For this use case, Databricks does not have built-in triggers directly tied to Delta table write operations, as seen in traditional databases. However, you can achieve this functionality using one of the following approaches:Approach 1: File ...
- 0 kudos
- 22222 Views
- 3 replies
- 7 kudos
Resolved! collect_list by preserving order based on another variable - Spark SQL
I am using databricks sql notebook to run these queries. I have a Python UDF like %python from pyspark.sql.functions import udf from pyspark.sql.types import StringType, DoubleType, DateType def get_sell_price(sale_prices): return sale_...
- 22222 Views
- 3 replies
- 7 kudos
- 7 kudos
I had a similar situation where I was trying to order the days of the week from Monday to Sunday. I saw solutions that use Python but was wanting to do it all in SQL. My original attempt was to use: CONCAT_WS(',', COLLECT_LIST(DISTINCT t.LOAD_ORIG_...
- 7 kudos
- 15858 Views
- 4 replies
- 4 kudos
Workflow timeout
Always set a timeout for your jobs! It not only safeguards against unforeseen hang-ups but also optimizes resource utilization. Equally essential is to consider having a threshold warning. This can alert you before a potential failure, allowing proac...
- 15858 Views
- 4 replies
- 4 kudos
- 4 kudos
We already have a policy and users are using clusters created with those to run their jobs. Since the policies are not based on job compute but on Power user compute, I am not able to set the job timeout_seconds.
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 585 | |
| 349 | |
| 287 |
