Hi,I have data in file like belowI have different types of row in my input file, column number 8 defines the type of the record.In the above file we have 4 types of records 00 to 03My requirement is:There will be multiple files in the source path, ea...
Hi,I have data in file like below I have different types of row in my input file, column number 8 defines the type of the record.In the above file we have 4 types of records 00 to 03My requirement is:There will be multiple files in the source path, e...
Hi @Retired_mod ,If i run again with same files sometimes records will be missed from same files of the previous run or records will be missed from different file.Example:run1: 1 record missing in file1, no issue with other filesrun2: 1 record missin...
Hola all, I'm trying to download all the dashboard definitions, however, I can only download the folder structure with no file inside. The procedure I'm using is* Go to the dashboard folder* Download it as DBC or source archiveUnfortunately, the DBC ...
Hello Everyone, I have build this script in order to collect ganglia metrics but the size of stderr and sdtout ganglia is 0. It doesn't work. I Have put this script on Workspace due to migration databricks all init-script should be place on Workspace...
Hi, Is there any error you are getting? Also, please tag @Debayan with your next comment which will notify me. Thanks!
I've been working on obtaining DDL at the schema level in Hive Metastore within GCP-hosted Databricks. I've implemented a Python code that generates SQL files in the dbfs/temp directory. However, when running the code, I'm encountering a "file path n...
Hi, the error code snippet with the whole error may help to determine the issue, also, considering the above points may also work as a fix.
Hi Everyone, am unable to see the permission button in sql warehouse to provide access to other users.I have admin rights and databricks is premium subscription.
Hi, Could you please provide a screenshot of the SQL warehouse? Also, you can go through: https://docs.databricks.com/en/security/auth-authz/access-control/sql-endpoint-acl.htmlAlso, please tag @Debayan with your next comment which will notify me. Th...
Hi,Anyone knows how I'm able to monitor cost of the SQL Serverless? I'm using Databricks in Azure and I'm not sure where to find cost generated by compute resources hosted on Databricks.
Hi, You can calculate the pricing in https://www.databricks.com/product/pricing/databricks-sql also, https://azure.microsoft.com/en-in/pricing/details/databricks/#:~:text=Sign%20in%20to%20the%20Azure,asked%20questions%20about%20Azure%20pricing. For A...
I am trying to create an external table using csv file which is stored in ADLS gen2 My account owner has created a storage credential and an external location I am a databricks user who all privileges on external location when trying to create a tabl...
Hi @Shubhanshu , Could you please try https://learn.microsoft.com/en-us/answers/questions/1314651/error-invalid-configuration-value-detected-for-fs and see if this is working?
I have a strange issue with Databricks notebooks and Google Colab notebooks, where I cannot get the results from requests library that would be consistent to what I get on my local computer. Dir Surveys (wyo.gov) ("http://pipeline.wyo.gov/r_Direction...
Additional information: when I download html file from the above mentioned website, I still get not the same html page as I get from the same code that I run on my computer or any other computer. The size of downloaded html file is the same as I get ...
Creating an application to capture cluster metrics and sending HTTP REST request to the Spark History Server's API endpoint to retrieve a list of applications. This request doesn't generate logs in the Spark History Server's log files. The Spark Hist...
Hi,We need to monitor Databricks jobs and we have made a setup where are able to get the prometheus metrics, however, we are lagging an overview of which metrics refer to what.Namely, we need to monitor the following:failed jobs : is a job failedtabl...
I have reposted this post in "Administation and Architecture"Monitoring job metrics - Databricks - 42956
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<s3-bucket-name>" ] }, { "Effect": "Allow", "Action": [ "s3:Pu...
at_lst = ['131','132','133'] at_tup = (*at_lst,) print(at_tup) ('131','132','133')<div> <div><span>In my sql query, i am trying to pass this on a parameter, however, it doesn't work. <div> <div><div><div><span>%sql<br /><div><span>select * from ma...
@Retired_mod I am writing sql using the magic command in the cell block, `%%sql`. Is there a way to pass a parameter in the query without using the `execute` method of the cursor object? Can you please share an example?
Hi Community,I am looking for a way to access the Power BI tables from databricks and import them as a spark dataframe into my databricks notebook.As far as I have seen, there is a Power BI connector to load data from databricks into Power BI but not...
I'm using StarRocks Connector[2] to ingest data to StarRocks on DataBricks 13.1 (powered by Spark 3.4.0). The connector could run on community Spark 3.4, but fail on the DBR. The reason is (the full stack trace is attached)java.lang.IncompatibleClass...
