- 7778 Views
- 8 replies
- 1 kudos
Resolved! How to Use the CharIndex with Databricks SQL
When applying the following T-SQL I don't get any errors on MS SQL ServerSELECT DISTINCT * FROM dbo.account LEFT OUTER JOIN dbo.crm2cburl_lookup ON account.Id = CRM2CBURL_Lookup.[Key] LEFT OUTER JOIN dbo.organizations ON CRM2CBURL_Lookup.CB_UR...
- 7778 Views
- 8 replies
- 1 kudos
- 19868 Views
- 4 replies
- 3 kudos
Resolved! install a custom python package from azure devops artifact to databricks cluster
I am trying to install a package which was uploaded into the azure devops artifact into the databricks cluster by using pip.conf. Basically below are the steps I followed.(step 1 : install in local IDE)Uploaded the package to azure devops feed using ...
- 19868 Views
- 4 replies
- 3 kudos
- 3 kudos
Thanks for your time @Debayan Mukherjee​ and @Kaniz Fatma​ . We have figured out the issue along with the infra team that we had to do a public ip whitelisting of the databricks clusters in azure.I have checked the ip adress from the Spark cluster U...
- 3 kudos
- 32031 Views
- 4 replies
- 8 kudos
AnalysisException: is not a Delta table.
Hello, I changed the DBR from 7.2 to 10.4 and I receive the following error: AnalysisException: is not a Delta table. The table is create , using DELTA. so for sure is a Delta table, even though, I read that I read that from vers. 8 all tables are De...
- 32031 Views
- 4 replies
- 8 kudos
- 8 kudos
Hi @JOSELITA MOLTISANTI​ can you run the following commands and share the output? table_name = "stg_data_load" path = spark.sql(f"describe detail {table_name}").select("location").collect()[0][0].replace('dbfs:', '') dbutils.fs.ls(path)
- 8 kudos
- 2711 Views
- 1 replies
- 0 kudos
Resolved! DLT Pipelines call same table
Orchestration of when DLT runs is handled by Azure Data Factory. There are scenario's a table within a DLT pipeline needs to run on a different schedule.Is there a pipeline configuration option to be set to allow the same table to be ran by two diff...
- 2711 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Kristian Foster​ , It should not be possible. Every pipeline owns its table and multiple pipelines cannot write to the same table.
- 0 kudos
- 13442 Views
- 6 replies
- 9 kudos
Resolved! How do I kick off Azure Data Factory from within Databricks?
I want to kick off ingestion in ADF from Databricks. When ADF ingestion is done, my DBX bronze-silver-gold pipeline follows within DBX.I see it is possible to call Databricks notebooks from ADF. Can I also go the other way? I want to start the ingest...
- 13442 Views
- 6 replies
- 9 kudos
- 5434 Views
- 1 replies
- 5 kudos
"Databricks" - "PySpark" - Read "JSON" file - Azure Blob container - "APPEND BLOB"
Hi All, We are getting JSON files in Azure blob container and its "Blob Type" is "Append Blob".We are getting an error "AnalysisException: Unable to infer schema for JSON. It must be specified manually.", when we try to read using below mentioned scr...
- 5434 Views
- 1 replies
- 5 kudos
- 5 kudos
There currently does not appear to be direct support for append blob reads, however, converting the append blob to block blob [and then parquet or delta, etc.] are a viable option:https://kb.databricks.com/en_US/data-sources/wasb-check-blob-types?_ga...
- 5 kudos
- 2445 Views
- 2 replies
- 0 kudos
Resolved! Question regarding ZORDER option of OPTIMIZE
Is the order of the columns in ZORDER important? For example, does ZORDER BY (product, site) and ZORDER BY (site, product) produce the same results?
- 2445 Views
- 2 replies
- 0 kudos
- 4181 Views
- 2 replies
- 6 kudos
Resolved! Is it a good idea to use a managed delta table as a temporal table?
Hi all!I would like to use a managed delta table as a temporal table, meaning:to create a managed table in the middle of ETL processto drop the managed table right after the processThis way I can perform merge, insert, or delete oprations better than...
- 4181 Views
- 2 replies
- 6 kudos
- 6 kudos
@Kwangwon Yi​ Instead of performance, main issue with managed table is whenever you delete table, data under that table gets deleted.If you have good use case on Reporting, best approach is to go with external storage location to store your managed t...
- 6 kudos
- 3419 Views
- 1 replies
- 1 kudos
Resolved! why aren't rdds using all available cores of executor?
I'm extracting data from a custom format by day of month using a 32 core executor. I'm using rdds to distribute work across cores of the executor. I'm seeing an intermittent issue where for a run sometimes I see 31 cores being used as expected and ot...
- 3419 Views
- 1 replies
- 1 kudos
- 1 kudos
I may have figured this out! I'm explicitly setting the number of slices instead of using the default.days_rdd = sc.parallelize(days_to_process,len(days_to_process))
- 1 kudos
- 3385 Views
- 2 replies
- 3 kudos
Count on External Table to Azure Data Storage is taking too long
I have created an External table to Azure Data Lake Storage Gen2.The Container has about 200K Json files.The structure of the json files are created with```CREATE EXTERNAL TABLE IF NOT EXISTS dbo.table( ComponentInfo STRUCT<ComponentHost: STRING, ...
- 3385 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Ena Vu​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 3 kudos
- 3330 Views
- 3 replies
- 1 kudos
Unable to update permissions in Unity Catalog object in Single User Mode DBR 11.2
We're trying to update permissions of catalogs in Single User Cluster Mode but running into following error We were able to update permission in Shared Mode. We used Shared mode to create objects but using single user mode to update permission seems...
- 3330 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Parth Salvi​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 1 kudos
- 5465 Views
- 4 replies
- 0 kudos
Temporarily disable Photon
Is it possible to temporarily disable Photon?I have a large workload that greatly benefits from Photon apart from a specific operation therein that is actually slowed by Photon. It's not worth creating a separate cluster for this operation however, s...
- 5465 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Aaron Morgan​ Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thank...
- 0 kudos
- 12242 Views
- 4 replies
- 0 kudos
Resolved! String converstion to datetimestamp format
Hello i'm converting hana sql code in databricks. we have 4 columns all in string format, start date, start time, end date, endtime..1) what expression i can use to convert values of startdate & start time from string format to datetimeformat wit...
- 12242 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello Mattconcat & to_timstamp function partially worked, values with 24 timestamp format not converted. any other approach i can think .?
- 0 kudos
- 4617 Views
- 3 replies
- 1 kudos
Resolved! Unable to create SQL Warehouse using AZURE DATABRICKS Subscription
Dear Team,I am unable to create a SQL Warehouse using AZURE Databricks subscription , below are the detailsi am able to create a cluster with single node ,but i am not able to create SQL Warehouse , i am using Cluster size - 2X Small , as part of the...
- 4617 Views
- 3 replies
- 1 kudos
- 1 kudos
In portal.azure.com, write quotas and ask for an increase. https://portal.azure.com/#view/Microsoft_Azure_Capacity/QuotaMenuBlade/~/myQuotasAdditionally, in SQL warehouse "Advanced options," you can change "Spot instance policy" from "cost-optimized"...
- 1 kudos
- 7820 Views
- 1 replies
- 3 kudos
Resolved! Generate new token error
Hello. I need to install Databricks CLI. While I am trying to generate new access token (User Settings->Generate new token) I get the following error:Could not create token with comment "cli" and lifetime (seconds) of 86400.I tried with different com...
- 7820 Views
- 1 replies
- 3 kudos
- 3 kudos
Please check in the Admin console that tokens are enabled and that you can manage them.
- 3 kudos
| User | Count |
|---|---|
| 1638 | |
| 791 | |
| 536 | |
| 349 | |
| 287 |
